settingsLogin | Registersettings

[openstack-dev] [Murano] Heat resource isolation withing single stack

0 votes

Hi Everyone,

While looking through Heat templates generation code in Murano I've
realized it has a major design flaw: there is no isolation between Heat
resources generated by different apps.

Every app manifest can access and modify its environment stack in any way.
For example it can delete instances and other resources belonging to other
applications. This may be not so bad for Murano 0.4 but it becomes critical
for AppCatalog (0.5) as there is no trust relations between applications
and it may be unacceptable that untrusted application can gain complete
write access over the whole stack.

There is also a problem of name collisions - resources generated by
different applications may have the same names. This is especially probable
between resources generated by different instances of the same app. This
also affects Parameters/Output of Heat templates as each application
instance must generate unique names for them (and do not forget them later
as they are needed to read output results).

I think we need at least to know how we going to solve it before 0.5

Here is possible directions i can think of:

  1. Use nested Heat stacks. I'm not sure it solves naming collisions and
    that nested stacks can have their own Output

  2. Control all stack template modifications and track which resource was
    created by which app. Give applications read-only access to resources they
    don't own

  3. Auto-generate resource names. Auto-add prefixes/suffixes to
    resource/output etc names indicating owning app instance ID and remove them
    upon read access from workflow so that generated names would be invisible
    to workflow. That would also mean all VMs would have generated names

Hope to see better ideas and suggestions in this thread

--
Sincerely yours
Stanislav (Stan) Lagun
Senior Developer
Mirantis
35b/3, Vorontsovskaya St.
Moscow, Russia
Skype: stanlagun
www.mirantis.com
slagun at mirantis.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.openstack.org/pipermail/openstack-dev/attachments/20140221/b1cee117/attachment.html

asked Feb 21, 2014 in openstack-dev by Stan_Lagun (4,160 points)   1 2 2
retagged Feb 25, 2015 by admin

4 Responses

0 votes

On Fri, Feb 21, 2014 at 06:37:27PM +0400, Stan Lagun wrote:
Hi Everyone,

While looking through Heat templates generation code in Murano I've
realized it has a major design flaw: there is no isolation between Heat
resources generated by different apps.

Can you define the requirement for "isolation" in more detail? Are you
referring simply to namespace isolation, or do you need auth level
isolation, e.g something enforced via keystone?

Every app manifest can access and modify its environment stack in any way.
For example it can delete instances and other resources belonging to other
applications. This may be not so bad for Murano 0.4 but it becomes critical
for AppCatalog (0.5) as there is no trust relations between applications
and it may be unacceptable that untrusted application can gain complete
write access over the whole stack.

All requests to Heat are scoped by tenant/project, so unless you enforce
resource-level access policy (which we sort-of started looking at with
OS::Heat::AccessPolicy), this is expected behavior.

There is also a problem of name collisions - resources generated by
different applications may have the same names. This is especially probable
between resources generated by different instances of the same app. This
also affects Parameters/Output of Heat templates as each application
instance must generate unique names for them (and do not forget them later
as they are needed to read output results).

A heirarchy of nested stacks, with each application defined as a separate
stack seems the obvious solution here.

I think we need at least to know how we going to solve it before 0.5

Here is possible directions i can think of:

  1. Use nested Heat stacks. I'm not sure it solves naming collisions and
    that nested stacks can have their own Output

I think it does, and yes all stacks can have their own outputs, including
nested stacks.

Of particular interest to you may be the provider resource interface to
nested stacks, which will allow you to define (via a series of nested stack
templates) custom resource types defining each of your applications.

See this old blog post, which will give you the providers/environments 101,
and contains links to most of the related heat docs:

http://hardysteven.blogspot.co.uk/2013/10/heat-providersenvironments-101-ive.html

  1. Control all stack template modifications and track which resource was
    created by which app. Give applications read-only access to resources they
    don't own

I think we need more info on the use-case here, but perhaps you can either
use the AccessPolicy resource, or we can work on defining an enhanced
version which meets your requirements.

  1. Auto-generate resource names. Auto-add prefixes/suffixes to
    resource/output etc names indicating owning app instance ID and remove them
    upon read access from workflow so that generated names would be invisible
    to workflow. That would also mean all VMs would have generated names

Heat already does this internally, we create unique names for all your
instances, unless you explicitly provide a name via the OS::Nova::Server
"name" property.

It might help if you could provide a really simplified example of the
problem you are facing, or links to the real templates which we could
review and make suggestions?

Steve

responded Feb 21, 2014 by Steven_Hardy (16,900 points)   2 7 12
0 votes

Steve, thank you for very valuable suggestions. Your block post is really
great - I've read about environments in Heat documentation but didn't
really understood them until now.

Usage of nested stacks may or may not solve my problem depending on what is
possible to do within those stacks.
Let me explain with simple example.

As you probably know Murano uses Heat for all infrastructure-related
operations. This means if some application from Catalog needs VM instance
or any other type of OpenStack resource it creates it by inserting a
snippet into user's Heat stack template and executes UPDATE STACK command.

Now suppose there is WordPress application published in App Catalog.
WordPress app manifest says that it requires installation of MySql. There
is also another application in AppCatalog called GaleraMySql that is known
to be compatible with MySql. In Murano Dashboard user creates new
environment (this corresponds to Heat stack and is not related to what is
called environment in Heat)
and puts WordPress and GaleraMySql on it. Then he connects them so that
GaleraMySql instance would be used in WordPress for MySql requirement.

WordPress and GaleraMySql were developed by different vendors that are not
aware of each others presence. But because of unfortunate combination of
circumstances both vendors chose to merge exactly the same snippet into
user's stack:

"Resources": {
    "myHost": {
        "Type": "AWS::EC2::Instance",
        "Properties": {
            "InstanceType": "large",
            "ImageId": "someImage"
        }
    }
}

Then instead of 2 different VMs there would be only one. Things would be
even worse if there was already resource "myHost" in user's stack.
It is more than a name-collision problem as incorrectly written application
manifest can cause any imaginable harm to the stack.

The obvious solution would be to give each app dedicated nested stack and
restrict it to that nested stack only. This would be a best solution. All I
need is to have the same level of control on nested stack I have on outer
stack - get stack template, modify and update them, access output
attributes. Is it possible to retrieve nested stack template, modify it and
populate it back to Heat?

Another option would be create separate top-level stacks for each app. But
in Murano applications themselves composed of smaller parts and in practice
this would lead to creation of dozen stacks with most of them containing
single resource. And then we would have to implement transaction update
between several stacks, coordinated deletion etc. This would also be bad
from a user's point of view at he doesn't expect to find long list of
stacks he has no idea where they came from.

My other options were on how nested stacks can be emulated on top of single
stack by controlling which app created which resource and dynamically
adjust resource names back and forth ("myHost" in example above) to some
unique values in a way that is opaque to application

On Fri, Feb 21, 2014 at 8:20 PM, Steven Hardy wrote:

On Fri, Feb 21, 2014 at 06:37:27PM +0400, Stan Lagun wrote:

Hi Everyone,

While looking through Heat templates generation code in Murano I've
realized it has a major design flaw: there is no isolation between Heat
resources generated by different apps.

Can you define the requirement for "isolation" in more detail? Are you
referring simply to namespace isolation, or do you need auth level
isolation, e.g something enforced via keystone?

Every app manifest can access and modify its environment stack in any
way.
For example it can delete instances and other resources belonging to
other
applications. This may be not so bad for Murano 0.4 but it becomes
critical
for AppCatalog (0.5) as there is no trust relations between applications
and it may be unacceptable that untrusted application can gain complete
write access over the whole stack.

All requests to Heat are scoped by tenant/project, so unless you enforce
resource-level access policy (which we sort-of started looking at with
OS::Heat::AccessPolicy), this is expected behavior.

There is also a problem of name collisions - resources generated by
different applications may have the same names. This is especially
probable
between resources generated by different instances of the same app. This
also affects Parameters/Output of Heat templates as each application
instance must generate unique names for them (and do not forget them
later
as they are needed to read output results).

A heirarchy of nested stacks, with each application defined as a separate
stack seems the obvious solution here.

I think we need at least to know how we going to solve it before 0.5

Here is possible directions i can think of:

  1. Use nested Heat stacks. I'm not sure it solves naming collisions and
    that nested stacks can have their own Output

I think it does, and yes all stacks can have their own outputs, including
nested stacks.

Of particular interest to you may be the provider resource interface to
nested stacks, which will allow you to define (via a series of nested stack
templates) custom resource types defining each of your applications.

See this old blog post, which will give you the providers/environments 101,
and contains links to most of the related heat docs:

http://hardysteven.blogspot.co.uk/2013/10/heat-providersenvironments-101-ive.html

  1. Control all stack template modifications and track which resource was
    created by which app. Give applications read-only access to resources
    they
    don't own

I think we need more info on the use-case here, but perhaps you can either
use the AccessPolicy resource, or we can work on defining an enhanced
version which meets your requirements.

  1. Auto-generate resource names. Auto-add prefixes/suffixes to
    resource/output etc names indicating owning app instance ID and remove
    them
    upon read access from workflow so that generated names would be invisible
    to workflow. That would also mean all VMs would have generated names

Heat already does this internally, we create unique names for all your
instances, unless you explicitly provide a name via the OS::Nova::Server
"name" property.

It might help if you could provide a really simplified example of the
problem you are facing, or links to the real templates which we could
review and make suggestions?

Steve


OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Sincerely yours
Stanislav (Stan) Lagun
Senior Developer
Mirantis
35b/3, Vorontsovskaya St.
Moscow, Russia
Skype: stanlagun
www.mirantis.com
slagun at mirantis.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL:

responded Feb 21, 2014 by Stan_Lagun (4,160 points)   1 2 2
0 votes

On 22/02/14 03:37, Stan Lagun wrote:
Hi Everyone,

While looking through Heat templates generation code in Murano I've
realized it has a major design flaw: there is no isolation between
Heat resources generated by different apps.

Every app manifest can access and modify its environment stack in any
way. For example it can delete instances and other resources belonging
to other applications. This may be not so bad for Murano 0.4 but it
becomes critical for AppCatalog (0.5) as there is no trust relations
between applications and it may be unacceptable that untrusted
application can gain complete write access over the whole stack.

There is also a problem of name collisions - resources generated by
different applications may have the same names. This is especially
probable between resources generated by different instances of the
same app. This also affects Parameters/Output of Heat templates as
each application instance must generate unique names for them (and do
not forget them later as they are needed to read output results).

I think we need at least to know how we going to solve it before 0.5

Here is possible directions i can think of:

  1. Use nested Heat stacks. I'm not sure it solves naming collisions
    and that nested stacks can have their own Output

  2. Control all stack template modifications and track which resource
    was created by which app. Give applications read-only access to
    resources they don't own

  3. Auto-generate resource names. Auto-add prefixes/suffixes to
    resource/output etc names indicating owning app instance ID and remove
    them upon read access from workflow so that generated names would be
    invisible to workflow. That would also mean all VMs would have
    generated names

Hope to see better ideas and suggestions in this thread

I would like the Murano project to evaluate the upcoming heat
software-config resources to represent app configuration.
Some patches are still in review [1][2] and I expect more changes before
feature freeze, but now would be a good time for Murano to get involved
to validate that software-config meets your needs.

The current implementation of the SoftwareDeployment resource actually
has the same issue mentioned above; any authenticated user can currently
create a deployment which results in software being deployed to any
server in that tenant/project. There may be times where this is useful,
but there will soon be a default policy enforced which ensures that the
deployment resource and the server resource belong to the same stack.

Inferring from the above, it sounds like Murano is transforming an app
definition into a single flat heat template. Using a combination of
resource providers and software config should give you a significantly
cleaner model, and you may even find that some of the problems you're
solving in Murano are now solved in the Heat layer.

[1]
https://review.openstack.org/#/q/status:open+topic:bp/hot-software-config-rest,n,z
[2]
https://review.openstack.org/#/q/status:open+topic:bp/hot-software-config,n,z

responded Feb 23, 2014 by Steve_Baker (7,380 points)   1 3 6
0 votes

I would like the Murano project to evaluate the upcoming heat
software-config resources to represent app configuration.

Yes, we are closely watching progress being made in Heat in that area and
evaluating how all new great features can be used in Murano

Inferring from the above, it sounds like Murano is transforming an app
definition into a single flat heat template.

It is just not that simple. Murano by itself neither generates any
templates nor deploys applications. Murano is just App Catalog.

Application definitions that Catalog consists of are more then just a
declarations. They also contains imperative executable parts written in
Murano DSL language.
Murano itself can be viewed as a runtime for application manifests.

Murano engine provides carefully designed APIs for DSL code to access Heat
and other OpenStack services. Those APIs are more restrictive and safe than
native OpenStack APIs (provided by various python-clients) and DSL cannot
access OpenStack services by any means other than those APIs. Heat API is
one of them. With this API DSL can access Heat stack and make some changes
in it.

It is up to Murano users (software publishers) who fill AppCatalog to
decide how and if to use Heat. They are free to use Heat just for
infrastructure-level management and do software config by some external
tools or use new software-config in Heat or even directly talk to Nova,
Trove etc.
But no matter how application decides to implement it, Murano must
guarantee that DSL code of one app cannot damage or compromise OpenStack
resources that belongs to another app. So even if new Heat feature makes
life easier we still need to guarantee safe behavior of DSL when using old
Heat template syntax or older Heat version that doesn't have that feature.

On Sun, Feb 23, 2014 at 11:50 PM, Steve Baker wrote:

On 22/02/14 03:37, Stan Lagun wrote:

Hi Everyone,

While looking through Heat templates generation code in Murano I've
realized it has a major design flaw: there is no isolation between
Heat resources generated by different apps.

Every app manifest can access and modify its environment stack in any
way. For example it can delete instances and other resources belonging
to other applications. This may be not so bad for Murano 0.4 but it
becomes critical for AppCatalog (0.5) as there is no trust relations
between applications and it may be unacceptable that untrusted
application can gain complete write access over the whole stack.

There is also a problem of name collisions - resources generated by
different applications may have the same names. This is especially
probable between resources generated by different instances of the
same app. This also affects Parameters/Output of Heat templates as
each application instance must generate unique names for them (and do
not forget them later as they are needed to read output results).

I think we need at least to know how we going to solve it before 0.5

Here is possible directions i can think of:

  1. Use nested Heat stacks. I'm not sure it solves naming collisions
    and that nested stacks can have their own Output

  2. Control all stack template modifications and track which resource
    was created by which app. Give applications read-only access to
    resources they don't own

  3. Auto-generate resource names. Auto-add prefixes/suffixes to
    resource/output etc names indicating owning app instance ID and remove
    them upon read access from workflow so that generated names would be
    invisible to workflow. That would also mean all VMs would have
    generated names

Hope to see better ideas and suggestions in this thread

I would like the Murano project to evaluate the upcoming heat
software-config resources to represent app configuration.
Some patches are still in review [1][2] and I expect more changes before
feature freeze, but now would be a good time for Murano to get involved
to validate that software-config meets your needs.

The current implementation of the SoftwareDeployment resource actually
has the same issue mentioned above; any authenticated user can currently
create a deployment which results in software being deployed to any
server in that tenant/project. There may be times where this is useful,
but there will soon be a default policy enforced which ensures that the
deployment resource and the server resource belong to the same stack.

Inferring from the above, it sounds like Murano is transforming an app
definition into a single flat heat template. Using a combination of
resource providers and software config should give you a significantly
cleaner model, and you may even find that some of the problems you're
solving in Murano are now solved in the Heat layer.

[1]

https://review.openstack.org/#/q/status:open+topic:bp/hot-software-config-rest,n,z
[2]

https://review.openstack.org/#/q/status:open+topic:bp/hot-software-config,n,z


OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Sincerely yours
Stanislav (Stan) Lagun
Senior Developer
Mirantis
35b/3, Vorontsovskaya St.
Moscow, Russia
Skype: stanlagun
www.mirantis.com
slagun at mirantis.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL:

responded Feb 23, 2014 by Stan_Lagun (4,160 points)   1 2 2
...