settingsLogin | Registersettings

[openstack-dev] [cross-project][infra][keystone] Moving towards a Identity v3-only on Devstack - Next Steps

0 votes

Hi folks,

Although the Identity v2 API is deprecated as of Mitaka [1], some services
haven't implemented proper support to v3 yet. For instance, we implemented
a patch that made DevStack v3 by default that, when merged, broke a lot of
project gates in a few hours [2]. This happened due to specific services in
compatibility issues with Keystone v3 API, such as hardcoded v2 usage,
usage of removed keystoneclient CLI, requesting v2 service tokens and the
lack of keystoneauth session usage.

To discuss those points, we did a cross-project work session in the Newton
Summit[3]. One point we are working on at this momment is creating gates to
ensure the main OpenStack services can live without the Keystone v2 API. Those
gates setup devstack with only Identity v3 enabled and run the Tempest suite
on this environment.

We already did that for a few services, like Nova, Cinder, Glance, Neutron,
Swift. We are doing the same job for other services such as Ironic, Magnum,
Ceilometer, Heat and Barbican [4].

In addition, we are creating jobs to run functional tests for the services
on this identity v3-only environment[5]. Also, we have a couple of other
fronts that we are doing like removing some hardcoded v2 usage [6],
implementing keystoneauth sessions support in clients and APIs [7].

Our plan is to keep tackling as many items from the cross-project session
etherpad as we can, so we can achieve more confidence in moving to a DevStack
working v3-only, making sure everyone is prepared to work with Keystone v3
API.

Feedbacks and reviews are very appreciated.

[1] https://review.openstack.org/#/c/251530/
[2] https://etherpad.openstack.org/p/v3-only-devstack
[3] https://etherpad.openstack.org/p/newton-keystone-v3-devstack
[4]
https://review.openstack.org/#/q/project:openstack-infra/project-config+branch:master+topic:v3-only-integrated
[5] https://review.openstack.org/#/q/topic:v3-only-functionals-tests-gates
[6] https://review.openstack.org/#/q/topic:remove-hardcoded-keystone-v2
[7] https://review.openstack.org/#/q/topic:use-ksa

Cheers,

Raildo


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked May 12, 2016 in openstack-dev by Raildo_Mascena (1,060 points)   2
retagged Jan 25, 2017 by admin

11 Responses

0 votes

We just had to revert another v3 "fix" because it wasn't verified to
work correctly in the gate - https://review.openstack.org/#/c/315631/

While I realize project-config patches are harder to test, you can do so
with a bogus devstack-gate change that has the same impact in some cases
(like the case above).

I think the important bit on moving forward is that every patch here
which might be disruptive has some manual verification about it working
posted in review by v3 team members before we approve them.

I also think we need to largely stay non voting on the v3 only job until
we're quite confident that the vast majority of things are flipped over
(for instance there remains an issue in nova <=> ironic communication
with v3 last time I looked). That allows us to fix things faster because
we don't wedge some slice of the projects in a gate failure.

-Sean

On 05/12/2016 11:08 AM, Raildo Mascena wrote:
Hi folks,

Although the Identity v2 API is deprecated as of Mitaka [1], some
services haven't implemented proper support to v3 yet. For instance,
we implemented a patch that made DevStack v3 by default that, when
merged, broke a lot of project gates in a few hours [2]. This
happened due to specific services incompatibility issues with Keystone
v3 API, such as hardcoded v2 usage, usage of removed keystoneclient CLI,
requesting v2 service tokens and the lack of keystoneauth session usage.

To discuss those points, we did a cross-project work
session in the Newton Summit[3]. One point we are working on at this
momment is creating gates to ensure the main OpenStack services
can live without the Keystone v2 API. Those gates setup devstack with
only Identity v3 enabled and run the Tempest suite on this environment.

We already did that for a few services, like Nova, Cinder, Glance,
Neutron, Swift. We are doing the same job for other services such
as Ironic, Magnum, Ceilometer, Heat and Barbican [4].

In addition, we are creating jobs to run functional tests for the
services on this identity v3-only environment[5]. Also, we have a couple
of other fronts that we are doing like removing some hardcoded v2 usage
[6], implementing keystoneauth sessions support in clients and APIs [7].

Our plan is to keep tackling as many items from the cross-project
session etherpad as we can, so we can achieve more confidence in moving
to a DevStack working v3-only, making sure everyone is prepared to work
with Keystone v3 API.

Feedbacks and reviews are very appreciated.

[1] https://review.openstack.org/#/c/251530/
[2] https://etherpad.openstack.org/p/v3-only-devstack
[3] https://etherpad.openstack.org/p/newton-keystone-v3-devstack
[4] https://review.openstack.org/#/q/project:openstack-infra/project-config+branch:master+topic:v3-only-integrated
[5] https://review.openstack.org/#/q/topic:v3-only-functionals-tests-gates
[6] https://review.openstack.org/#/q/topic:remove-hardcoded-keystone-v2
[7] https://review.openstack.org/#/q/topic:use-ksa

Cheers,

Raildo


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Sean Dague
http://dague.net


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 12, 2016 by Sean_Dague (66,200 points)   4 8 14
0 votes

On Thu, May 12, 2016 at 10:42 AM, Sean Dague sean@dague.net wrote:

We just had to revert another v3 "fix" because it wasn't verified to
work correctly in the gate - https://review.openstack.org/#/c/315631/

While I realize project-config patches are harder to test, you can do so
with a bogus devstack-gate change that has the same impact in some cases
(like the case above).

I think the important bit on moving forward is that every patch here
which might be disruptive has some manual verification about it working
posted in review by v3 team members before we approve them.

I also think we need to largely stay non voting on the v3 only job until
we're quite confident that the vast majority of things are flipped over
(for instance there remains an issue in nova <=> ironic communication
with v3 last time I looked). That allows us to fix things faster because
we don't wedge some slice of the projects in a gate failure.

    -Sean

On 05/12/2016 11:08 AM, Raildo Mascena wrote:

Hi folks,

Although the Identity v2 API is deprecated as of Mitaka [1], some
services haven't implemented proper support to v3 yet. For instance,
we implemented a patch that made DevStack v3 by default that, when
merged, broke a lot of project gates in a few hours [2]. This
happened due to specific services incompatibility issues with Keystone
v3 API, such as hardcoded v2 usage, usage of removed keystoneclient CLI,
requesting v2 service tokens and the lack of keystoneauth session usage.

To discuss those points, we did a cross-project work
session in the Newton Summit[3]. One point we are working on at this
momment is creating gates to ensure the main OpenStack services
can live without the Keystone v2 API. Those gates setup devstack with
only Identity v3 enabled and run the Tempest suite on this environment.

We already did that for a few services, like Nova, Cinder, Glance,
Neutron, Swift. We are doing the same job for other services such
as Ironic, Magnum, Ceilometer, Heat and Barbican [4].

In addition, we are creating jobs to run functional tests for the
services on this identity v3-only environment[5]. Also, we have a couple
of other fronts that we are doing like removing some hardcoded v2 usage
[6], implementing keystoneauth sessions support in clients and APIs [7].

Our plan is to keep tackling as many items from the cross-project
session etherpad as we can, so we can achieve more confidence in moving
to a DevStack working v3-only, making sure everyone is prepared to work
with Keystone v3 API.

Feedbacks and reviews are very appreciated.

[1] https://review.openstack.org/#/c/251530/
[2] https://etherpad.openstack.org/p/v3-only-devstack
[3] https://etherpad.openstack.org/p/newton-keystone-v3-devstack
[4]
https://review.openstack.org/#/q/project:openstack-infra/project-config+branch:master+topic:v3-only-integrated
[5]
https://review.openstack.org/#/q/topic:v3-only-functionals-tests-gates
[6] https://review.openstack.org/#/q/topic:remove-hardcoded-keystone-v2
[7] https://review.openstack.org/#/q/topic:use-ksa

Cheers,

Raildo

This also comes back to the conversation at the summit. We need to propose
the timeline to turn over for V3 (regardless of voting/non-voting today) so
that it is possible to set the timeline that is expected for everything to
get fixed (and where we are expecting/planning to stop reverting while
focusing on fixing the v3-only changes).

I am going to ask the Keystone team to set forth the timeline and commit to
getting the pieces in order so that we can make v3-only voting rather than
playing the propose/revert game we're currently doing. A proposed timeline
and gameplan will only help at this point.

--Morgan


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 12, 2016 by Morgan_Fainberg (17,320 points)   2 4 9
0 votes

On 05/12/2016 01:47 PM, Morgan Fainberg wrote:
This also comes back to the conversation at the summit. We need to
propose the timeline to turn over for V3 (regardless of
voting/non-voting today) so that it is possible to set the timeline that
is expected for everything to get fixed (and where we are
expecting/planning to stop reverting while focusing on fixing the
v3-only changes).

I am going to ask the Keystone team to set forth the timeline and commit
to getting the pieces in order so that we can make v3-only voting rather
than playing the propose/revert game we're currently doing. A proposed
timeline and gameplan will only help at this point.

A timeline would be good (proposed below), but there are also other bits
of the approach we should consider.

I would expect, for instance,
gate-tempest-dsvm-neutron-identity-v3-only-full to be on keystone, and
it does not appear to be. Is there a reason why?

With that on keystone, devstack-gate, devstack, tempest the integrated
space should be pretty well covered. There really is no need to also go
stick this on glance, nova, cinder, neutron, swift I don't think,
because they only really use keystone through pretty defined interfaces.

Then some strategic use of nv jobs on things we know would have some
additional interactions here (because we know they are currently broken
or they do interesting things) like ironic, heat, trove, would probably
be pretty useful.

That starts building up the list of known breaks the keystone folks are
tracking, which should get a drum beat every week in email about
outstanding issues, and issues fixed.

The goal of gate-tempest-dsvm-neutron-identity-v3-only-full should not
be for that to be voting, ever. It should be to use that as a good
indicator that changing the default in devstack (and thus in the
majority of upstream jobs) to not ever enable v2.

Because of how v3 support exists in projects (largely hidden behind
keystoneauth), it is really unlikely to rando regress once fixed. There
just aren't that many knobs a project has that would make that happen.
So I think we can make forward progress without a voting backstop until
we get to a point where we can just throw the big red switch (with
warning) on a Monday (probably early in the Otaca cycle) and say there
you go. It's now the project job to handle it. And they'll all get fair
warning for the month prior to the big red switch.

-Sean

--
Sean Dague
http://dague.net


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 12, 2016 by Sean_Dague (66,200 points)   4 8 14
0 votes

On Thu, May 12, 2016 at 3:19 PM Sean Dague sean@dague.net wrote:

On 05/12/2016 01:47 PM, Morgan Fainberg wrote:

This also comes back to the conversation at the summit. We need to
propose the timeline to turn over for V3 (regardless of
voting/non-voting today) so that it is possible to set the timeline that
is expected for everything to get fixed (and where we are
expecting/planning to stop reverting while focusing on fixing the
v3-only changes).

I am going to ask the Keystone team to set forth the timeline and commit
to getting the pieces in order so that we can make v3-only voting rather
than playing the propose/revert game we're currently doing. A proposed
timeline and gameplan will only help at this point.

A timeline would be good (proposed below), but there are also other bits
of the approach we should consider.

I would expect, for instance,
gate-tempest-dsvm-neutron-identity-v3-only-full to be on keystone, and
it does not appear to be. Is there a reason why?

We made this here: https://review.openstack.org/#/c/311169/

With that on keystone, devstack-gate, devstack, tempest the integrated
space should be pretty well covered. There really is no need to also go
stick this on glance, nova, cinder, neutron, swift I don't think,
because they only really use keystone through pretty defined interfaces.

Then some strategic use of nv jobs on things we know would have some
additional interactions here (because we know they are currently broken
or they do interesting things) like ironic, heat, trove, would probably
be pretty useful.

That starts building up the list of known breaks the keystone folks are
tracking, which should get a drum beat every week in email about
outstanding issues, and issues fixed.

++ Sounds a good Idea, I'll work to make this happen.

The goal of gate-tempest-dsvm-neutron-identity-v3-only-full should not
be for that to be voting, ever. It should be to use that as a good
indicator that changing the default in devstack (and thus in the
majority of upstream jobs) to not ever enable v2.

We intend use this job as a indicator to find bugs related to this, but the
idea to make this gate-tempest-dsvm-neutron-identity-v3-only-full voting is
make sure that anyone are sending anything v3 incompatible.

Because of how v3 support exists in projects (largely hidden behind

keystoneauth), it is really unlikely to rando regress once fixed. There
just aren't that many knobs a project has that would make that happen.
So I think we can make forward progress without a voting backstop until
we get to a point where we can just throw the big red switch (with
warning) on a Monday (probably early in the Otaca cycle) and say there
you go. It's now the project job to handle it. And they'll all get fair
warning for the month prior to the big red switch.

    -Sean

--
Sean Dague
http://dague.net


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 12, 2016 by Raildo_Mascena (1,060 points)   2
0 votes

On 05/12/2016 01:47 PM, Morgan Fainberg wrote:

On Thu, May 12, 2016 at 10:42 AM, Sean Dague <sean@dague.net
sean@dague.net> wrote:

We just had to revert another v3 "fix" because it wasn't verified to
work correctly in the gate - https://review.openstack.org/#/c/315631/

While I realize project-config patches are harder to test, you can
do so
with a bogus devstack-gate change that has the same impact in some
cases
(like the case above).

I think the important bit on moving forward is that every patch here
which might be disruptive has some manual verification about it
working
posted in review by v3 team members before we approve them.

I also think we need to largely stay non voting on the v3 only job
until
we're quite confident that the vast majority of things are flipped
over
(for instance there remains an issue in nova <=> ironic communication
with v3 last time I looked). That allows us to fix things faster
because
we don't wedge some slice of the projects in a gate failure.

        -Sean

On 05/12/2016 11:08 AM, Raildo Mascena wrote:
> Hi folks,
>
> Although the Identity v2 API is deprecated as of Mitaka [1], some
> services haven't implemented proper support to v3 yet. For instance,
> we implemented a patch that made DevStack v3 by default that, when
> merged, broke a lot of project gates in a few hours [2]. This
> happened due to specific services incompatibility issues with
Keystone
> v3 API, such as hardcoded v2 usage, usage of removed
keystoneclient CLI,
> requesting v2 service tokens and the lack of keystoneauth
session usage.
>
> To discuss those points, we did a cross-project work
> session in the Newton Summit[3]. One point we are working on at this
> momment is creating gates to ensure the main OpenStack services
> can live without the Keystone v2 API. Those gates setup devstack
with
> only Identity v3 enabled and run the Tempest suite on this
environment.
>
> We already did that for a few services, like Nova, Cinder, Glance,
> Neutron, Swift. We are doing the same job for other services such
> as Ironic, Magnum, Ceilometer, Heat and Barbican [4].
>
> In addition, we are creating jobs to run functional tests for the
> services on this identity v3-only environment[5]. Also, we have
a couple
> of other fronts that we are doing like removing some hardcoded
v2 usage
> [6], implementing keystoneauth sessions support in clients and
APIs [7].
>
> Our plan is to keep tackling as many items from the cross-project
> session etherpad as we can, so we can achieve more confidence in
moving
> to a DevStack working v3-only, making sure everyone is prepared
to work
> with Keystone v3 API.
>
> Feedbacks and reviews are very appreciated.
>
> [1] https://review.openstack.org/#/c/251530/
> [2] https://etherpad.openstack.org/p/v3-only-devstack
> [3] https://etherpad.openstack.org/p/newton-keystone-v3-devstack
> [4]
https://review.openstack.org/#/q/project:openstack-infra/project-config+branch:master+topic:v3-only-integrated
> [5]
https://review.openstack.org/#/q/topic:v3-only-functionals-tests-gates
> [6]
https://review.openstack.org/#/q/topic:remove-hardcoded-keystone-v2
> [7] https://review.openstack.org/#/q/topic:use-ksa
>
> Cheers,
>
> Raildo
>
>
>

This also comes back to the conversation at the summit. We need to
propose the timeline to turn over for V3 (regardless of
voting/non-voting today) so that it is possible to set the timeline
that is expected for everything to get fixed (and where we are
expecting/planning to stop reverting while focusing on fixing the
v3-only changes).

I am going to ask the Keystone team to set forth the timeline and
commit to getting the pieces in order so that we can make v3-only
voting rather than playing the propose/revert game we're currently
doing. A proposed timeline and gameplan will only help at this point.

I would like to draw a line in the sand and say that it has to be there
for Ocata. We should be working through the issues the Newton release,
and have a firm "Ocata should expect to be run V3 only, with V2.0 an
optional feature that can be enabled for backwards compatibility if
required."

To me, Ocata is the finish line: there have been a lot of features that
Keystone has needed for a long time, and they are finally starting to
come together. V3 support, and all that it enables is on of the big
ones, and we need to give people enough information to plan, and enough
time to adjust.

V3 support is essential for the way that most people are deploying: User
database coming from an external source like LDAP, service users stored
locally. We need to treat this as baseline, and make sure that the
deployment tools understand this and can make it happen in their toolchains.

--Morgan


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 12, 2016 by Adam_Young (19,940 points)   2 7 9
0 votes

On 12/05/2016 1:47 PM, Morgan Fainberg wrote:

On Thu, May 12, 2016 at 10:42 AM, Sean Dague <sean@dague.net
sean@dague.net> wrote:

We just had to revert another v3 "fix" because it wasn't verified to
work correctly in the gate - https://review.openstack.org/#/c/315631/

While I realize project-config patches are harder to test, you can do so
with a bogus devstack-gate change that has the same impact in some cases
(like the case above).

I think the important bit on moving forward is that every patch here
which might be disruptive has some manual verification about it working
posted in review by v3 team members before we approve them.

I also think we need to largely stay non voting on the v3 only job until
we're quite confident that the vast majority of things are flipped over
(for instance there remains an issue in nova <=> ironic communication
with v3 last time I looked). That allows us to fix things faster because
we don't wedge some slice of the projects in a gate failure.

         -Sean

On 05/12/2016 11:08 AM, Raildo Mascena wrote:
 > Hi folks,
 >
 > Although the Identity v2 API is deprecated as of Mitaka [1], some
 > services haven't implemented proper support to v3 yet. For instance,
 > we implemented a patch that made DevStack v3 by default that, when
 > merged, broke a lot of project gates in a few hours [2]. This
 > happened due to specific services incompatibility issues with
Keystone
 > v3 API, such as hardcoded v2 usage, usage of removed
keystoneclient CLI,
 > requesting v2 service tokens and the lack of keystoneauth session
usage.
 >
 > To discuss those points, we did a cross-project work
 > session in the Newton Summit[3]. One point we are working on at this
 > momment is creating gates to ensure the main OpenStack services
 > can live without the Keystone v2 API. Those gates setup devstack with
 > only Identity v3 enabled and run the Tempest suite on this
environment.
 >
 > We already did that for a few services, like Nova, Cinder, Glance,
 > Neutron, Swift. We are doing the same job for other services such
 > as Ironic, Magnum, Ceilometer, Heat and Barbican [4].
 >
 > In addition, we are creating jobs to run functional tests for the
 > services on this identity v3-only environment[5]. Also, we have a
couple
 > of other fronts that we are doing like removing some hardcoded v2
usage
 > [6], implementing keystoneauth sessions support in clients and
APIs [7].
 >
 > Our plan is to keep tackling as many items from the cross-project
 > session etherpad as we can, so we can achieve more confidence in
moving
 > to a DevStack working v3-only, making sure everyone is prepared
to work
 > with Keystone v3 API.
 >
 > Feedbacks and reviews are very appreciated.
 >
 > [1] https://review.openstack.org/#/c/251530/
 > [2] https://etherpad.openstack.org/p/v3-only-devstack
 > [3] https://etherpad.openstack.org/p/newton-keystone-v3-devstack
 > [4]
https://review.openstack.org/#/q/project:openstack-infra/project-config+branch:master+topic:v3-only-integrated
 > [5]
https://review.openstack.org/#/q/topic:v3-only-functionals-tests-gates
 > [6]
https://review.openstack.org/#/q/topic:remove-hardcoded-keystone-v2
 > [7] https://review.openstack.org/#/q/topic:use-ksa
 >
 > Cheers,
 >
 > Raildo
 >
 >
 >

This also comes back to the conversation at the summit. We need to
propose the timeline to turn over for V3 (regardless of
voting/non-voting today) so that it is possible to set the timeline that
is expected for everything to get fixed (and where we are
expecting/planning to stop reverting while focusing on fixing the
v3-only changes).

I am going to ask the Keystone team to set forth the timeline and commit
to getting the pieces in order so that we can make v3-only voting rather
than playing the propose/revert game we're currently doing. A proposed
timeline and gameplan will only help at this point.

can anyone confirm when we deprecated keystonev2? i see a bp[1] related
to deprecation that was 'implemented' in 2013.

i realise switching to v3 breaks many gates but it'd be good to at some
point say it's not 'keystonev3 breaking the gate' but rather 'projectx
is breaking the gate because they are using keystonev2 which was
deprecated 4 cycles ago'. given the deprecation period allowed already,
can we say "here's some help, fix/merge this by
, or your gate will be broken until then"?
(assuming all the above items by Raildo doesn't fix everything).

[1] https://blueprints.launchpad.net/keystone/+spec/deprecate-v2-api

cheers,

--
gord


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 12, 2016 by gordon_chung (19,300 points)   2 3 8
0 votes

On 05/12/2016 06:39 PM, gordon chung wrote:

On 12/05/2016 1:47 PM, Morgan Fainberg wrote:

On Thu, May 12, 2016 at 10:42 AM, Sean Dague <sean@dague.net
sean@dague.net> wrote:

 We just had to revert another v3 "fix" because it wasn't verified to
 work correctly in the gate - https://review.openstack.org/#/c/315631/

 While I realize project-config patches are harder to test, you can do so
 with a bogus devstack-gate change that has the same impact in some cases
 (like the case above).

 I think the important bit on moving forward is that every patch here
 which might be disruptive has some manual verification about it working
 posted in review by v3 team members before we approve them.

 I also think we need to largely stay non voting on the v3 only job until
 we're quite confident that the vast majority of things are flipped over
 (for instance there remains an issue in nova <=> ironic communication
 with v3 last time I looked). That allows us to fix things faster because
 we don't wedge some slice of the projects in a gate failure.

          -Sean

 On 05/12/2016 11:08 AM, Raildo Mascena wrote:
  > Hi folks,
  >
  > Although the Identity v2 API is deprecated as of Mitaka [1], some
  > services haven't implemented proper support to v3 yet. For instance,
  > we implemented a patch that made DevStack v3 by default that, when
  > merged, broke a lot of project gates in a few hours [2]. This
  > happened due to specific services incompatibility issues with
 Keystone
  > v3 API, such as hardcoded v2 usage, usage of removed
 keystoneclient CLI,
  > requesting v2 service tokens and the lack of keystoneauth session
 usage.
  >
  > To discuss those points, we did a cross-project work
  > session in the Newton Summit[3]. One point we are working on at this
  > momment is creating gates to ensure the main OpenStack services
  > can live without the Keystone v2 API. Those gates setup devstack with
  > only Identity v3 enabled and run the Tempest suite on this
 environment.
  >
  > We already did that for a few services, like Nova, Cinder, Glance,
  > Neutron, Swift. We are doing the same job for other services such
  > as Ironic, Magnum, Ceilometer, Heat and Barbican [4].
  >
  > In addition, we are creating jobs to run functional tests for the
  > services on this identity v3-only environment[5]. Also, we have a
 couple
  > of other fronts that we are doing like removing some hardcoded v2
 usage
  > [6], implementing keystoneauth sessions support in clients and
 APIs [7].
  >
  > Our plan is to keep tackling as many items from the cross-project
  > session etherpad as we can, so we can achieve more confidence in
 moving
  > to a DevStack working v3-only, making sure everyone is prepared
 to work
  > with Keystone v3 API.
  >
  > Feedbacks and reviews are very appreciated.
  >
  > [1] https://review.openstack.org/#/c/251530/
  > [2] https://etherpad.openstack.org/p/v3-only-devstack
  > [3] https://etherpad.openstack.org/p/newton-keystone-v3-devstack
  > [4]
 https://review.openstack.org/#/q/project:openstack-infra/project-config+branch:master+topic:v3-only-integrated
  > [5]
 https://review.openstack.org/#/q/topic:v3-only-functionals-tests-gates
  > [6]
 https://review.openstack.org/#/q/topic:remove-hardcoded-keystone-v2
  > [7] https://review.openstack.org/#/q/topic:use-ksa
  >
  > Cheers,
  >
  > Raildo
  >
  >
  >

This also comes back to the conversation at the summit. We need to
propose the timeline to turn over for V3 (regardless of
voting/non-voting today) so that it is possible to set the timeline that
is expected for everything to get fixed (and where we are
expecting/planning to stop reverting while focusing on fixing the
v3-only changes).

I am going to ask the Keystone team to set forth the timeline and commit
to getting the pieces in order so that we can make v3-only voting rather
than playing the propose/revert game we're currently doing. A proposed
timeline and gameplan will only help at this point.

can anyone confirm when we deprecated keystonev2? i see a bp[1] related
to deprecation that was 'implemented' in 2013.

i realise switching to v3 breaks many gates but it'd be good to at some
point say it's not 'keystonev3 breaking the gate' but rather 'projectx
is breaking the gate because they are using keystonev2 which was
deprecated 4 cycles ago'. given the deprecation period allowed already,
can we say "here's some help, fix/merge this by
, or your gate will be broken until then"?
(assuming all the above items by Raildo doesn't fix everything).

I'd like to say Ocata.

[1] https://blueprints.launchpad.net/keystone/+spec/deprecate-v2-api

cheers,


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 12, 2016 by Adam_Young (19,940 points)   2 7 9
0 votes

On Thu, May 12, 2016 at 6:39 PM, gordon chung gord@live.ca wrote:

can anyone confirm when we deprecated keystonev2? i see a bp[1] related
to deprecation that was 'implemented' in 2013.

We did this last release, see the sixth bullet point here:

http://docs.openstack.org/releasenotes/keystone/mitaka.html#deprecation-notes

I also vocalized this on the regular, -dev, and -ops mailing list.

i realise switching to v3 breaks many gates but it'd be good to at some
point say it's not 'keystonev3 breaking the gate' but rather 'projectx
is breaking the gate because they are using keystonev2 which was
deprecated 4 cycles ago'. given the deprecation period allowed already,
can we say "here's some help, fix/merge this by
, or your gate will be broken until then"?
(assuming all the above items by Raildo doesn't fix everything).

[1] https://blueprints.launchpad.net/keystone/+spec/deprecate-v2-api

cheers,

--
gord


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 13, 2016 by s.martinelli_at_gmai (5,460 points)   1 2 2
0 votes

On 05/12/2016 08:49 PM, Steve Martinelli wrote:

On Thu, May 12, 2016 at 6:39 PM, gordon chung <gord@live.ca
gord@live.ca> wrote:

can anyone confirm when we deprecated keystonev2? i see a bp[1] related
to deprecation that was 'implemented' in 2013.

We did this last release, see the sixth bullet point here:

http://docs.openstack.org/releasenotes/keystone/mitaka.html#deprecation-notes

I also vocalized this on the regular, -dev, and -ops mailing list.

I might suggest moving that to the preface portion of the release notes,
as that's probably one of the most important bits of info to people.

-Sean

--
Sean Dague
http://dague.net


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 13, 2016 by Sean_Dague (66,200 points)   4 8 14
0 votes

On 13 May 2016 at 04:15, Sean Dague sean@dague.net wrote:

On 05/12/2016 01:47 PM, Morgan Fainberg wrote:

This also comes back to the conversation at the summit. We need to
propose the timeline to turn over for V3 (regardless of
voting/non-voting today) so that it is possible to set the timeline that
is expected for everything to get fixed (and where we are
expecting/planning to stop reverting while focusing on fixing the
v3-only changes).

I am going to ask the Keystone team to set forth the timeline and commit
to getting the pieces in order so that we can make v3-only voting rather
than playing the propose/revert game we're currently doing. A proposed
timeline and gameplan will only help at this point.

A timeline would be good (proposed below), but there are also other bits
of the approach we should consider.

That was my job to get sent to the TC. I'll get on it.

I would expect, for instance,
gate-tempest-dsvm-neutron-identity-v3-only-full to be on keystone, and
it does not appear to be. Is there a reason why?

To test that keystone works with keystone v3? Otherwise what you're doing
is making it so that keystone's gate breaks every time neutron does
something that's not v3 compatible which brings it to our attention but
otherwise just gets in the way. The hope was to push the check job failure
back to the service so that its not purely keystone's job to run around and
fix all the other services when the incompatible change is discovered.

With that on keystone, devstack-gate, devstack, tempest the integrated
space should be pretty well covered. There really is no need to also go
stick this on glance, nova, cinder, neutron, swift I don't think,
because they only really use keystone through pretty defined interfaces.

Initially i would have agreed, and there has been a voting job on devstack
with keystone v3 only that proves that all of these services can work
together for at least a cycle. Where we got stung was all the plugins and
configuration options used in these services that don't get tested by that
integrated gate job. The hope was that by pushing these jobs out to the
services we would get more coverage of the service specific configurations
- but I can see that might not be working.

Then some strategic use of nv jobs on things we know would have some
additional interactions here (because we know they are currently broken
or they do interesting things) like ironic, heat, trove, would probably
be pretty useful.

That starts building up the list of known breaks the keystone folks are
tracking, which should get a drum beat every week in email about
outstanding issues, and issues fixed.

The goal of gate-tempest-dsvm-neutron-identity-v3-only-full should not
be for that to be voting, ever. It should be to use that as a good
indicator that changing the default in devstack (and thus in the
majority of upstream jobs) to not ever enable v2.

Because of how v3 support exists in projects (largely hidden behind
keystoneauth), it is really unlikely to rando regress once fixed. There
just aren't that many knobs a project has that would make that happen.
So I think we can make forward progress without a voting backstop until
we get to a point where we can just throw the big red switch (with
warning) on a Monday (probably early in the Otaca cycle) and say there
you go. It's now the project job to handle it. And they'll all get fair
warning for the month prior to the big red switch.

I agree. Very early in the Otaca cycle is also the timeframe we had
discussed at summit so it looks like there is a good consensus there and
i'll get that proposal to TC this week.

For now we maintain the v3-only jobs as non-voting and we continue to push
the changes particularly to projects that are not tested in the default
devstack integrated gate test.

PS. I assume i'm right in assuming it's just impossible/infeasable to have
project-config changes determine all the jobs that are affected by a change
and run those as the project-config gate. It seems like one of the last few
places where we can commit something that breaks everyone and have never
noticed.

    -Sean

--
Sean Dague
http://dague.net


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 16, 2016 by jamielennox_at_gmail (1,700 points)   1
...