settingsLogin | Registersettings

[openstack-dev] [cinder] [nova] os-brick privsep failures and an upgrade strategy?

0 votes

os-brick 1.4 was released over the weekend, and was the first os-brick
to include privsep. We got a really odd failure rate in the
grenade-multinode jobs (1/3 - 1/2) after wards which was super non
obvious why. Hemma looks to have figured it out (this is a summary of
what I've seen on IRC to pull it all together)

Remembering the following -
https://github.com/openstack-dev/grenade#theory-of-upgrade and
https://governance.openstack.org/reference/tags/assert_supports-upgrade.html#requirements
- New code must work with N-1 configs. So this is master running with
mitaka configuration.

privsep requires a sudo rule or rootwrap rule (to get to sudo) to allow
the privsep daemon to be spawned for volume actions.

During gate testing we have a blanket sudoer rule for the stack user
during the run of grenade.sh. It has to do system level modifications
broadly to perform the upgrade. This sudoer rule is deleted at the end
of the grenade.sh run before Tempest tests are run, so that Tempest
tests don't accidentally require root privs on their target environment.

Grenade also makes sure that some resources live across the upgrade
boundary. This includes a boot from volume guest, which is torn down
before testing starts. And this is where things get interesting.

This means there is a volume teardown needed before grenade ends. But
there is only one. In single node grenade this happens about 30 seconds
for the end of the script, triggers the privsep daemon start, and then
we're done. And the 50stacksh sudoers file is removed. In multinode,
if the boot from volume server is on the upgrade node, then the same
thing happens. However, if it instead ended up on the subnode, which
is not upgraded, then the volume tear down in on the old node. No
os-brick calls are made on the upgraded node before grenade finishes.
The 50stacksh sudoers file is removed, as expected.

And now all volume tests on those nodes fail.

Which is what should happen. The point is that in production no one is
going to put a blanket sudoers rule like that in place. It's just we
needed it for this activity, and the userid on the services being the
same as the shell user (which is not root) let this fallback rule be used.

The crux of the problem is that os-brick 1.4 and privsep can't be used
without a config file change during the upgrade. Which violates our
policy, because it breaks rolling upgrades.

So... we have a few options:

1) make an exception here with release notes, because it's the only way
to move forward.

2) have some way for os-brick to use either mode for a transition period
(depending on whether privsep is configured to work)

3) Something else.... ?

https://bugs.launchpad.net/os-brick/+bug/1592043 is the bug we've got on
this. We should probably sort out the path forward here on the ML as
there are a bunch of folks in a bunch of different time zones that have
important perspectives here.

-Sean

--
Sean Dague
http://dague.net


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Jun 14, 2016 in openstack-dev by Sean_Dague (66,200 points)   4 8 14
retagged Jan 26, 2017

20 Responses

0 votes

On Tue, Jun 14, 2016 at 07:49:54AM -0400, Sean Dague wrote:

[snip]

The crux of the problem is that os-brick 1.4 and privsep can't be used
without a config file change during the upgrade. Which violates our
policy, because it breaks rolling upgrades.

os-vif support is going to face exactly the same problem. We just followed
os-brick's lead by adding a change to devstack to explicitly set the
required config options in nova.conf to change privsep to use rootwrap
instead of plain sudo.

Basically every single user of privsep is likely to face the same
problem.

So... we have a few options:

1) make an exception here with release notes, because it's the only way
to move forward.

That's quite user hostile I think.

2) have some way for os-brick to use either mode for a transition period
(depending on whether privsep is configured to work)

I'm not sure that's viable - at least for os-vif we started from
a clean slate to assume use of privsep, so we won't be able to have
any optional fallback to non-privsep mode.

3) Something else.... ?

3) Add an API to oslo.privsep that lets us configure the default
command to launch the helper. Nova would invoke this on startup

  privsep.set_default_helper("sudo nova-rootwrap ....")

4) Have oslo.privsep install a sudo rule that grants permission
to run privsep-helper, without needing rootwrap.

5) Have each user of privsep install a sudo rule to grants
permission to run privsep-helper with just their specific
entry point context, without needing rootwrap

Any of 3/4/5 work out of the box, but I'm probably favouring
option 4, then 5, then 3.

Regards,
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jun 14, 2016 by Daniel_P._Berrange (27,920 points)   2 4 9
0 votes

On 06/14/2016 09:02 AM, Daniel P. Berrange wrote:
On Tue, Jun 14, 2016 at 07:49:54AM -0400, Sean Dague wrote:

[snip]

The crux of the problem is that os-brick 1.4 and privsep can't be used
without a config file change during the upgrade. Which violates our
policy, because it breaks rolling upgrades.

os-vif support is going to face exactly the same problem. We just followed
os-brick's lead by adding a change to devstack to explicitly set the
required config options in nova.conf to change privsep to use rootwrap
instead of plain sudo.

Basically every single user of privsep is likely to face the same
problem.

So... we have a few options:

1) make an exception here with release notes, because it's the only way
to move forward.

That's quite user hostile I think.

2) have some way for os-brick to use either mode for a transition period
(depending on whether privsep is configured to work)

I'm not sure that's viable - at least for os-vif we started from
a clean slate to assume use of privsep, so we won't be able to have
any optional fallback to non-privsep mode.

3) Something else.... ?

3) Add an API to oslo.privsep that lets us configure the default
command to launch the helper. Nova would invoke this on startup

  privsep.set_default_helper("sudo nova-rootwrap ....")

4) Have oslo.privsep install a sudo rule that grants permission
to run privsep-helper, without needing rootwrap.

5) Have each user of privsep install a sudo rule to grants
permission to run privsep-helper with just their specific
entry point context, without needing rootwrap

4 & 5 are the same as 1, because python packages don't have standardized
management of /etc in their infrastructure. The code can't roll forward
without a config change.

Option #3 is a new one, I wonder if that would get us past here better.

-Sean

--
Sean Dague
http://dague.net


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jun 14, 2016 by Sean_Dague (66,200 points)   4 8 14
0 votes

On 6/14/2016 11:33 AM, Sean Dague wrote:
On 06/14/2016 09:02 AM, Daniel P. Berrange wrote:

On Tue, Jun 14, 2016 at 07:49:54AM -0400, Sean Dague wrote:

[snip]

The crux of the problem is that os-brick 1.4 and privsep can't be used
without a config file change during the upgrade. Which violates our
policy, because it breaks rolling upgrades.

os-vif support is going to face exactly the same problem. We just followed
os-brick's lead by adding a change to devstack to explicitly set the
required config options in nova.conf to change privsep to use rootwrap
instead of plain sudo.

Basically every single user of privsep is likely to face the same
problem.

So... we have a few options:

1) make an exception here with release notes, because it's the only way
to move forward.

That's quite user hostile I think.

2) have some way for os-brick to use either mode for a transition period
(depending on whether privsep is configured to work)

I'm not sure that's viable - at least for os-vif we started from
a clean slate to assume use of privsep, so we won't be able to have
any optional fallback to non-privsep mode.

3) Something else.... ?

3) Add an API to oslo.privsep that lets us configure the default
command to launch the helper. Nova would invoke this on startup

  privsep.set_default_helper("sudo nova-rootwrap ....")

4) Have oslo.privsep install a sudo rule that grants permission
to run privsep-helper, without needing rootwrap.

5) Have each user of privsep install a sudo rule to grants
permission to run privsep-helper with just their specific
entry point context, without needing rootwrap

4 & 5 are the same as 1, because python packages don't have standardized
management of /etc in their infrastructure. The code can't roll forward
without a config change.

Option #3 is a new one, I wonder if that would get us past here better.

-Sean

Yeah #3 sounds the best to me, but would need to hear from Angus on this.

--

Thanks,

Matt Riedemann


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jun 14, 2016 by Matt_Riedemann (48,320 points)   3 7 21
0 votes

I just put up a WIP patch in os-brick that tests to see if os-privsep is
configured with
the helpercommand. If it's not, then os-brick falls back to using
processutils
with the root
helper and runasroot kwargs passed in.

https://review.openstack.org/#/c/329586
If you can check this out that would be helpful. If this is the route
we want to go,
then I'll add unit tests and take it out of WIP and try to get it in.

So, if nova.conf and cinder.conf aren't updated with the privseposbrick
sections
providing the helper
command, then osbrick will assume local
processutils calls
with the configured root
helper passed in.

This should be backwards compatible (grenade upgrade tests). But we
should encourage
admins to add that section to their nova.conf and cinder.conf files.
The other downside
to this is that if we have to keep this code in place, then we
effectively still have to maintain
rootwrap filters in place and keep them up to date. sadness

Walt

On 06/14/2016 04:49 AM, Sean Dague wrote:
os-brick 1.4 was released over the weekend, and was the first os-brick
to include privsep. We got a really odd failure rate in the
grenade-multinode jobs (1/3 - 1/2) after wards which was super non
obvious why. Hemma looks to have figured it out (this is a summary of
what I've seen on IRC to pull it all together)

Remembering the following -
https://github.com/openstack-dev/grenade#theory-of-upgrade and
https://governance.openstack.org/reference/tags/assert_supports-upgrade.html#requirements
- New code must work with N-1 configs. So this is master running with
mitaka configuration.

privsep requires a sudo rule or rootwrap rule (to get to sudo) to allow
the privsep daemon to be spawned for volume actions.

During gate testing we have a blanket sudoer rule for the stack user
during the run of grenade.sh. It has to do system level modifications
broadly to perform the upgrade. This sudoer rule is deleted at the end
of the grenade.sh run before Tempest tests are run, so that Tempest
tests don't accidentally require root privs on their target environment.

Grenade also makes sure that some resources live across the upgrade
boundary. This includes a boot from volume guest, which is torn down
before testing starts. And this is where things get interesting.

This means there is a volume teardown needed before grenade ends. But
there is only one. In single node grenade this happens about 30 seconds
for the end of the script, triggers the privsep daemon start, and then
we're done. And the 50stacksh sudoers file is removed. In multinode,
if the boot from volume server is on the upgrade node, then the same
thing happens. However, if it instead ended up on the subnode, which
is not upgraded, then the volume tear down in on the old node. No
os-brick calls are made on the upgraded node before grenade finishes.
The 50stacksh sudoers file is removed, as expected.

And now all volume tests on those nodes fail.

Which is what should happen. The point is that in production no one is
going to put a blanket sudoers rule like that in place. It's just we
needed it for this activity, and the userid on the services being the
same as the shell user (which is not root) let this fallback rule be used.

The crux of the problem is that os-brick 1.4 and privsep can't be used
without a config file change during the upgrade. Which violates our
policy, because it breaks rolling upgrades.

So... we have a few options:

1) make an exception here with release notes, because it's the only way
to move forward.

2) have some way for os-brick to use either mode for a transition period
(depending on whether privsep is configured to work)

3) Something else.... ?

https://bugs.launchpad.net/os-brick/+bug/1592043 is the bug we've got on
this. We should probably sort out the path forward here on the ML as
there are a bunch of folks in a bunch of different time zones that have
important perspectives here.

-Sean


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jun 14, 2016 by walter.boring_at_hpe (1,420 points)   1
0 votes

On Tue, 14 Jun 2016 at 23:04 Daniel P. Berrange berrange@redhat.com wrote:

On Tue, Jun 14, 2016 at 07:49:54AM -0400, Sean Dague wrote:

[snip]

Urgh, thanks for the in-depth analysis :/

The crux of the problem is that os-brick 1.4 and privsep can't be used

without a config file change during the upgrade. Which violates our
policy, because it breaks rolling upgrades.

os-vif support is going to face exactly the same problem. We just followed
os-brick's lead by adding a change to devstack to explicitly set the
required config options in nova.conf to change privsep to use rootwrap
instead of plain sudo.

Basically every single user of privsep is likely to face the same
problem.

So... we have a few options:

1) make an exception here with release notes, because it's the only way
to move forward.

That's quite user hostile I think.

2) have some way for os-brick to use either mode for a transition period
(depending on whether privsep is configured to work)

I'm not sure that's viable - at least for os-vif we started from
a clean slate to assume use of privsep, so we won't be able to have
any optional fallback to non-privsep mode.

3) Something else.... ?

3) Add an API to oslo.privsep that lets us configure the default
command to launch the helper. Nova would invoke this on startup

  privsep.set_default_helper("sudo nova-rootwrap ....")

4) Have oslo.privsep install a sudo rule that grants permission
to run privsep-helper, without needing rootwrap.

5) Have each user of privsep install a sudo rule to grants
permission to run privsep-helper with just their specific
entry point context, without needing rootwrap

Any of 3/4/5 work out of the box, but I'm probably favouring
option 4, then 5, then 3.

Yep (3) is quite possible, and the only reason it doesn't just do this
already is because there's no way to find the name of the rootwrap command
to use (from any library, privsep or os-brick) - and I was never very happy
with the current need to specify a command line in oslo.config purely for
this lame reason.

As Sean points out, all the others involve some sort of configuration
change preceding the code. I had imagined rollouts would work by pushing
out the harmless conf or sudoers change first, but hadn't appreciated the
strict change phases imposed by grenade (and ourselves).

If all "end-application" devs are happy calling something like (3) before
the first privileged operation occurs, then we should be good. I might
even take the opportunity to phrase it as a general privsep.init()
function, and then we can use it for any other top-of-main()
privilege-setup steps that need to be taken in the future.

responded Jun 14, 2016 by Angus_Lees (3,620 points)   2 3
0 votes

On 06/14/2016 06:11 PM, Angus Lees wrote:
Yep (3) is quite possible, and the only reason it doesn't just do this
already is because there's no way to find the name of the rootwrap
command to use (from any library, privsep or os-brick) - and I was never
very happy with the current need to specify a command line in
oslo.config purely for this lame reason.

As Sean points out, all the others involve some sort of configuration
change preceding the code. I had imagined rollouts would work by
pushing out the harmless conf or sudoers change first, but hadn't
appreciated the strict change phases imposed by grenade (and ourselves).

If all "end-application" devs are happy calling something like (3)
before the first privileged operation occurs, then we should be good. I
might even take the opportunity to phrase it as a general privsep.init()
function, and then we can use it for any other top-of-main()
privilege-setup steps that need to be taken in the future.

That sounds promising. It would be fine to emit a warning if it only was
using the default, asking people to make a configuration change to make
it go away. We're totally good with things functioning with warnings
after transitions, that ops can adjust during their timetable.

-Sean

--
Sean Dague
http://dague.net


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jun 15, 2016 by Sean_Dague (66,200 points)   4 8 14
0 votes

oslo.privsep change: https://review.openstack.org/#/c/329766/
And the nova change that uses it: https://review.openstack.org/#/c/329769

In particular I'm unsure if os-brick/os-vif is even loaded at this point in
nova-compute main(). Does anyone know when that actually happens or shall
I go exploring?

  • Gus

On Wed, 15 Jun 2016 at 11:43 Sean Dague sean@dague.net wrote:

On 06/14/2016 06:11 PM, Angus Lees wrote:

Yep (3) is quite possible, and the only reason it doesn't just do this
already is because there's no way to find the name of the rootwrap
command to use (from any library, privsep or os-brick) - and I was never
very happy with the current need to specify a command line in
oslo.config purely for this lame reason.

As Sean points out, all the others involve some sort of configuration
change preceding the code. I had imagined rollouts would work by
pushing out the harmless conf or sudoers change first, but hadn't
appreciated the strict change phases imposed by grenade (and ourselves).

If all "end-application" devs are happy calling something like (3)
before the first privileged operation occurs, then we should be good. I
might even take the opportunity to phrase it as a general privsep.init()
function, and then we can use it for any other top-of-main()
privilege-setup steps that need to be taken in the future.

That sounds promising. It would be fine to emit a warning if it only was
using the default, asking people to make a configuration change to make
it go away. We're totally good with things functioning with warnings
after transitions, that ops can adjust during their timetable.

    -Sean

--
Sean Dague
http://dague.net


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
--
Message protected by MailGuard: e-mail anti-virus, anti-spam and content
filtering.http://www.mailguard.com.au/mg
Click here to report this message as spam:
https://console.mailguard.com.au/ras/1ODUv4oqIN/4x80DVYpDOULTM59jB3mdH/0.82


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jun 15, 2016 by Angus_Lees (3,620 points)   2 3
0 votes

On 6/15/2016 1:11 AM, Angus Lees wrote:
oslo.privsep change: https://review.openstack.org/#/c/329766/
And the nova change that uses it: https://review.openstack.org/#/c/329769

In particular I'm unsure if os-brick/os-vif is even loaded at this point
in nova-compute main(). Does anyone know when that actually happens or
shall I go exploring?

  • Gus

An update on this.

The oslo.privsep change is merged and released in 1.9.0.

The nova change was updated to depend on oslo.privsep>=1.9.0.

To test that fix with os-brick 1.4.0, I made this change to requirements:

https://review.openstack.org/#/c/331885/

That depends on the nova fix and enables os-brick 1.4.0 which is the
version that uses oslo.privsep.

It also depends on a requirements change to stable/mitaka to enable
os-brick 1.4.0. I wasn't sure if this was necessary though given
upper-constraints for stable/mitaka already caps brick at 1.2.0.

Anyway, it still fails in the grenade multinode job even with the nova
and privsep fixes:

http://logs.openstack.org/85/331885/2/check/gate-grenade-dsvm-multinode/415e1bc/logs/new/screen-n-cpu.txt.gz?level=TRACE#_2016-06-21_16_21_31_015

And I confirmed via pip-freeze that it's using oslo.privsep 1.9.0 and
os-brick 1.4.0:

http://logs.openstack.org/85/331885/2/check/gate-grenade-dsvm-multinode/415e1bc/logs/pip2-freeze.txt.gz

I did verify that it's correctly pulling in the dependent nova fix:

http://logs.openstack.org/85/331885/2/check/gate-grenade-dsvm-multinode/415e1bc/logs/devstack-gate-setup-workspace-new.txt.gz#_2016-06-21_15_37_02_958

Angus, what should we be looking at from the privsep side for debugging
this?

--

Thanks,

Matt Riedemann


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jun 21, 2016 by Matt_Riedemann (48,320 points)   3 7 21
0 votes

On Wed, 22 Jun 2016 at 05:59 Matt Riedemann mriedem@linux.vnet.ibm.com
wrote:

Angus, what should we be looking at from the privsep side for debugging
this?

The line above the screen-n-cpu.txt.gz failure you linked to is:
2016-06-21 16:21:30.994

1840 WARNING oslo.privsep.daemon [-] privsep log:
/usr/local/bin/nova-rootwrap: Unauthorized command: privsep-helper
--config-file /etc/nova/nova.conf --privsepcontext
os
brick.privileged.default --privsepsockpath /tmp/tmpV5w2VC/privsep.sock
(no filter matched)

.. so nova-rootwrap is rejecting the privsep-helper command line because
no filter matched. This indicates the nova compute.filters file has not
been updated, or is incorrect.

As was later pointed out by mtreinish, grenade is attempting to run the
newton code against mitaka configs, and this includes using mitaka rootwrap
filters. Unfortunately, the change to add privsep to nova's rootwrap
filters wasn't approved until the newton cycle (so that all the os-brick
privsep-related changes could be approved together), and so this doesn't
Just Work.

Digging in further, it appears that there is a mechanism in grenade to
upgrade rootwrap filters between major releases, but this needs to be
explicitly updated for each project+release and hasn't been for
nova+mitaka->newton. I'm not sure how this is meant to work, since the
grenade "theory of upgrade" doesn't mention when configs should be updated
- the only mechanism provided is an "exception ... used sparingly."

Anyway, I added an upgrade step for nova mitaka->newton that updates
rootwrap filters appropriately(). Again, I'm not sure what this
communicates to deployers compared to cinder (which *did
have the updated
rootwrap filter merged in mitaka, but of course that update still needs to
be installed at some point).
(*) https://review.openstack.org/#/c/332610

responded Jun 22, 2016 by Angus_Lees (3,620 points)   2 3
0 votes

On 6/21/2016 10:12 PM, Angus Lees wrote:
On Wed, 22 Jun 2016 at 05:59 Matt Riedemann <mriedem@linux.vnet.ibm.com
mriedem@linux.vnet.ibm.com> wrote:

Angus, what should we be looking at from the privsep side for debugging
this?

The line above the screen-n-cpu.txt.gz failure you linked to is:
2016-06-21 16:21:30.994
1840
WARNING oslo.privsep.daemon [-] privsep log:
/usr/local/bin/nova-rootwrap: Unauthorized command: privsep-helper
--config-file /etc/nova/nova.conf --privsepcontext
os
brick.privileged.default --privsepsockpath
/tmp/tmpV5w2VC/privsep.sock (no filter matched)

.. so nova-rootwrap is rejecting the privsep-helper command line
because no filter matched. This indicates the nova compute.filters file
has not been updated, or is incorrect.

As was later pointed out by mtreinish, grenade is attempting to run the
newton code against mitaka configs, and this includes using mitaka
rootwrap filters. Unfortunately, the change to add privsep to nova's
rootwrap filters wasn't approved until the newton cycle (so that all the
os-brick privsep-related changes could be approved together), and so
this doesn't Just Work.

Digging in further, it appears that there is a mechanism in grenade to
upgrade rootwrap filters between major releases, but this needs to be
explicitly updated for each project+release and hasn't been for
nova+mitaka->newton. I'm not sure how this is meant to work, since
the grenade "theory of upgrade" doesn't mention when configs should be
updated - the only mechanism provided is an "exception ... used sparingly."

As noted in the review, my understanding of the config changes is
deprecation of options across release boundaries so that you can't drop
a config option that would break someone from release to release without
it being deprecated first. So deprecate option foo in mitaka, people
upgrading from liberty to mitaka aren't broken, but they get warnings in
mitaka so that when you drop the option in newton it's not a surprise
and consumers should have adjusted during mitaka.

For rootwrap filters I agree this is more complicated.

Anyway, I added an upgrade step for nova mitaka->newton that updates
rootwrap filters appropriately(). Again, I'm not sure what this
communicates to deployers compared to cinder (which *did
have the
updated rootwrap filter merged in mitaka, but of course that update
still needs to be installed at some point).
(*) https://review.openstack.org/#/c/332610

  • Gus


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Alternatively Walter had a potential workaround to fallback to rootwrap
for os-brick:

https://review.openstack.org/#/c/329586/

So we could maybe use that for newton. But os-vif doesn't have anything
like that, so we'd have to see what kind of (immediately deprecated)
workaround could happen for os-vif in newton and then drop that in ocata.

I'm told danpb is out until tomorrow though so we'll probably need to
wait to talk to him about options there.

--

Thanks,

Matt Riedemann


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jun 22, 2016 by Matt_Riedemann (48,320 points)   3 7 21
...