settingsLogin | Registersettings

[openstack-dev] [nova][trove] Protected openstack resources

0 votes

Several of us including Bruno Lago, Victoria Martinez de la Cruz (vkmc), Flavio Percoco, Nikhil Manchanda (SlickNik), Vipul Sabhaya (vipul), Doug Shelley (dougshelley66), and several others from the Trove team met in Vancouver and were joined by some others (who I do not know by name) from other projects. After the summit, I summarized that meeting in the email thread here [1http://markmail.org/message/cotket76zyhbvj34].

In that meeting, and in the course of other conversations, we concluded that projects like Trove require the ability to launch instances of resources from projects like Nova but bad things happen when a user directly interacts with these resources. It would therefore be highly advantageous to have a class of instances which are protected from direct user actions.

The "bad things" described above stem from the fact that the guest agent that Trove uses is a component that is on the guest instance and it communicates with the other Trove controller services over an oslo.messaging message queue. If a guest instance were compromised, the fact that it has a connection path to the message queue could become a vulnerability. Deployers of Trove have addressed these concerns and are able to operate a secure Trove system by launching Nova instances in a different tenant than the end user. The changes to Trove for this are currently not part of Trove but will be made available shortly.

Using oslo.messaging for the communication between Trove controller and the guest agents allows deployers to choose the underlying AMQP transport. However, oslo.messaging is tightly coupled with AMQP semantics. One proposed alternative (zaqar) that could address some of Trove's issues has no integration with oslo.messaging.

Therefore, to adopt zaqar, Trove would likely have to abandon oslo.messaging and integrate tightly with zaqar which strikes many of us as more restrictive and less attractive. I know of at least one user of Trove who has deployed oslo.messaging with qpid as the underlying transport, rather than the more commonly deployed RabbitMQ.

The request to create an oslo.messaging driver for zaqar (or was it a zaqar driver for oslo.messaging) met with some resistance for technical reasons. Flavio summarizes it in 2http://markmail.org/message/v3vm7bux5bi6vknu saying, "This is probably the main reason why there's no driver for Zaqar in oslo.messaging. That is, to prevent people from actually using Zaqar as a message bus in openstack."

Other projects like Sahara, and potentially others need a mechanism by which to protect their resources from direct manipulation by a user.

Several conversations ensued with members of Nova team and Bruno drafted a write-up summarizing some aspects of the problems. To facilitate a quick review of this request, the Trove team has put together a document and it is available for review at 3https://review.openstack.org/#/c/186357.

The request is to have Nova and potentially other OpenStack projects review the issues being described. They can then provide protected resources that projects like Trove can consume.

Equally, if you work on some other project that could benefit from protected resources, please chime in.

Please post comments on the request on the review (4https://review.openstack.org/#/c/186357) and register blueprints or work towards delivering these capabilities in your respective projects. The request is not prescriptive of how projects like Nova should implement these capabilities, it merely requests that they be created.

Thanks,

-amrith

[1] http://markmail.org/message/cotket76zyhbvj34
[2] http://markmail.org/message/v3vm7bux5bi6vknu
[3] https://review.openstack.org/#/c/186357
[4] https://review.openstack.org/#/c/186357


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Jun 3, 2015 in openstack-dev by Amrith_Kumar (10,500 points)   2 4 4

3 Responses

0 votes

On 3 June 2015 at 20:51, Amrith Kumar amrith@tesora.com wrote:
Several of us including Bruno Lago, Victoria Martinez de la Cruz (vkmc),
Flavio Percoco, Nikhil Manchanda (SlickNik), Vipul Sabhaya (vipul), Doug
Shelley (dougshelley66), and several others from the Trove team met in
Vancouver and were joined by some others (who I do not know by name) from
other projects. After the summit, I summarized that meeting in the email
thread here [1].

In that meeting, and in the course of other conversations, we concluded that
projects like Trove require the ability to launch instances of resources
from projects like Nova but bad things happen when a user directly interacts
with these resources. It would therefore be highly advantageous to have a
class of instances which are protected from direct user actions.

The “bad things” described above stem from the fact that the guest agent
that Trove uses is a component that is on the guest instance and it
communicates with the other Trove controller services over an oslo.messaging
message queue. If a guest instance were compromised, the fact that it has a
connection path to the message queue could become a vulnerability. Deployers
of Trove have addressed these concerns and are able to operate a secure
Trove system by launching Nova instances in a different tenant than the end
user. The changes to Trove for this are currently not part of Trove but will
be made available shortly.

FWIW, this is basically how Glance uses a multi-tenant Swift to store
images from Nova from various tenants.

I think there are more exciting ways that some folks have brewing that
involve some sort of combination of two tenants, or some such.

Using oslo.messaging for the communication between Trove controller and the
guest agents allows deployers to choose the underlying AMQP transport.
However, oslo.messaging is tightly coupled with AMQP semantics. One proposed
alternative (zaqar) that could address some of Trove’s issues has no
integration with oslo.messaging.

Therefore, to adopt zaqar, Trove would likely have to abandon oslo.messaging
and integrate tightly with zaqar which strikes many of us as more
restrictive and less attractive. I know of at least one user of Trove who
has deployed oslo.messaging with qpid as the underlying transport, rather
than the more commonly deployed RabbitMQ.

The request to create an oslo.messaging driver for zaqar (or was it a zaqar
driver for oslo.messaging) met with some resistance for technical reasons.
Flavio summarizes it in 2 saying, “This is probably the main reason why
there's no driver for Zaqar in oslo.messaging. That is, to prevent people
from actually using Zaqar as a message bus in openstack.”

So you could create a REST API for your agent to talk to. They are
quite well understood, but I have no idea about how your agent talks
to your server, so it could be a terrible idea.

Other projects like Sahara, and potentially others need a mechanism by which
to protect their resources from direct manipulation by a user.

Several conversations ensued with members of Nova team and Bruno drafted a
write-up summarizing some aspects of the problems. To facilitate a quick
review of this request, the Trove team has put together a document and it is
available for review at 3.

The request is to have Nova and potentially other OpenStack projects review
the issues being described. They can then provide protected resources that
projects like Trove can consume.

Equally, if you work on some other project that could benefit from protected
resources, please chime in.

Please post comments on the request on the review (4) and register
blueprints or work towards delivering these capabilities in your respective
projects. The request is not prescriptive of how projects like Nova should
implement these capabilities, it merely requests that they be created.

Why is the running your Nova VMs in a trove or sahara specific tenant
not good enough for your use case?

I am not trying to be difficult, I am just curious about what specific
issues something "better" would need to fix.

Thanks,
John


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jun 4, 2015 by John_Garbutt (15,460 points)   3 4 5
0 votes

John,

Thanks for your note. I've updated the review at https://review.openstack.org/#/c/186357/ with answers to some of your questions (and I added you to that review).

Trove's use-case like some of the other projects listed is different from Glance in that Trove has a guest agent. I've tried to explain that in more detail in patch set 5. I'd appreciate your comments.

Thanks,

-amrith

| -----Original Message-----
| From: John Garbutt [mailto:john@johngarbutt.com]
| Sent: Thursday, June 04, 2015 6:54 AM
| To: OpenStack Development Mailing List (not for usage questions)
| Subject: Re: [openstack-dev] [nova][trove] Protected openstack resources
|
| On 3 June 2015 at 20:51, Amrith Kumar amrith@tesora.com wrote:
| > Several of us including Bruno Lago, Victoria Martinez de la Cruz
| > (vkmc), Flavio Percoco, Nikhil Manchanda (SlickNik), Vipul Sabhaya
| > (vipul), Doug Shelley (dougshelley66), and several others from the
| > Trove team met in Vancouver and were joined by some others (who I do
| > not know by name) from other projects. After the summit, I summarized
| > that meeting in the email thread here [1].
| >
| > In that meeting, and in the course of other conversations, we
| > concluded that projects like Trove require the ability to launch
| > instances of resources from projects like Nova but bad things happen
| > when a user directly interacts with these resources. It would
| > therefore be highly advantageous to have a class of instances which are
| protected from direct user actions.
| >
| > The “bad things” described above stem from the fact that the guest
| > agent that Trove uses is a component that is on the guest instance and
| > it communicates with the other Trove controller services over an
| > oslo.messaging message queue. If a guest instance were compromised,
| > the fact that it has a connection path to the message queue could
| > become a vulnerability. Deployers of Trove have addressed these
| > concerns and are able to operate a secure Trove system by launching
| > Nova instances in a different tenant than the end user. The changes to
| > Trove for this are currently not part of Trove but will be made
| available shortly.
|
| FWIW, this is basically how Glance uses a multi-tenant Swift to store
| images from Nova from various tenants.
|
| I think there are more exciting ways that some folks have brewing that
| involve some sort of combination of two tenants, or some such.
|
| > Using oslo.messaging for the communication between Trove controller
| > and the guest agents allows deployers to choose the underlying AMQP
| transport.
| > However, oslo.messaging is tightly coupled with AMQP semantics. One
| > proposed alternative (zaqar) that could address some of Trove’s issues
| > has no integration with oslo.messaging.
| >
| > Therefore, to adopt zaqar, Trove would likely have to abandon
| > oslo.messaging and integrate tightly with zaqar which strikes many of
| > us as more restrictive and less attractive. I know of at least one
| > user of Trove who has deployed oslo.messaging with qpid as the
| > underlying transport, rather than the more commonly deployed RabbitMQ.
| >
| > The request to create an oslo.messaging driver for zaqar (or was it a
| > zaqar driver for oslo.messaging) met with some resistance for technical
| reasons.
| > Flavio summarizes it in 2 saying, “This is probably the main reason
| > why there's no driver for Zaqar in oslo.messaging. That is, to prevent
| > people from actually using Zaqar as a message bus in openstack.”
|
| So you could create a REST API for your agent to talk to. They are quite
| well understood, but I have no idea about how your agent talks to your
| server, so it could be a terrible idea.
|
| > Other projects like Sahara, and potentially others need a mechanism by
| > which to protect their resources from direct manipulation by a user.
| >
| > Several conversations ensued with members of Nova team and Bruno
| > drafted a write-up summarizing some aspects of the problems. To
| > facilitate a quick review of this request, the Trove team has put
| > together a document and it is available for review at 3.
| >
| > The request is to have Nova and potentially other OpenStack projects
| > review the issues being described. They can then provide protected
| > resources that projects like Trove can consume.
| >
| > Equally, if you work on some other project that could benefit from
| > protected resources, please chime in.
| >
| > Please post comments on the request on the review (4) and register
| > blueprints or work towards delivering these capabilities in your
| > respective projects. The request is not prescriptive of how projects
| > like Nova should implement these capabilities, it merely requests that
| they be created.
|
| Why is the running your Nova VMs in a trove or sahara specific tenant not
| good enough for your use case?
|
| I am not trying to be difficult, I am just curious about what specific
| issues something "better" would need to fix.
|
| Thanks,
| John
|
| __________________________________________________________________________
| OpenStack Development Mailing List (not for usage questions)
| Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
| http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jun 4, 2015 by Amrith_Kumar (10,500 points)   2 4 4
0 votes

Excerpts from Amrith Kumar's message of 2015-06-04 12:46:37 +0000:

John,

Thanks for your note. I've updated the review at https://review.openstack.org/#/c/186357/ with answers to some of your questions (and I added you to that review).

Trove's use-case like some of the other projects listed is different from Glance in that Trove has a guest agent. I've tried to explain that in more detail in patch set 5. I'd appreciate your comments.

We solved this in Akanda by placing the service VMs in a special
tenant, isolating them with security group rules, and then giving
the agent running in the VM a REST API connected to a private
management network owned by the same tenant that owns the VM. All
communication with the agent starts from a service on the outside,
through that management network. The VMs act as routers, so they
are also attached to the cloud-user's networks, but the agent doesn't
respond on those networks.

Doug


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jun 4, 2015 by Doug_Hellmann (87,520 points)   3 4 10
...