settingsLogin | Registersettings

Re: [openstack-dev] [tc] [trove][stable] proboscis tests randomly failing in stable branches; bug 1538506

0 votes

Matt,

We discussed this at the Trove meeting. Here's my current understanding of the situation:

  1. Your change https://review.openstack.org/#/c/290048/ which reverts the trove client side of the change should be merged.

  2. This change (the Trove API side) https://review.openstack.org/#/c/245845/ should also be reverted. I'm assuming that you'll submit a change for this as well for completeness.

  3. We need to blacklist python-troveclient 2.1.0 on Liberty, this is your change https://review.openstack.org/#/c/290021/

  4. We need to blacklist python-troveclient 2.1.0 on master, this is currently NOT what your change https://review.openstack.org/290168 does. I disagree with the approach of just increasing the minimum requirement from 1.2.0 to >=2.1.0.

  5. We need to request a new python-troveclient. Whether it should be called 2.1.1 or 2.2.0, I'm not positive. My thought is 2.1.1. But out of an abundance of caution, I'm going to look to #openstack-release/#openstack-infra for guidance on this.

Thanks,

-amrith

-----Original Message-----
From: Matt Riedemann [mailto:mriedem@linux.vnet.ibm.com]
Sent: Wednesday, March 09, 2016 12:42 PM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [trove][stable] proboscis tests randomly
failing in stable branches; bug 1538506

On 3/9/2016 8:54 AM, Amrith Kumar wrote:

Matt,

As I understand it, you have 4 changes up for review.

 Change [1] seeks to revert the change [6] to python-troveclient on
 master and reinstate the slave_of parameter.

 Change [2] is doing for stable/liberty, the same things that were
 done for master in [9] on master, and [7] on Kilo earlier.

 Change [3] is looking to blacklist python-troveclient 2.1.0 on
 stable/liberty.

 Change [4] is looking to bump the minimum python-troveclient

version
on master to 2.1.0.

Right, and there was actually another for that beat me to this:

https://review.openstack.org/#/c/290115/

Here are three questions I have:

(1) Wouldn't backward compatibility also require that you revert the
other change that removed slaveof on the server [5] ? After all, just
like a python client that calls create() with slave
of, a REST client
could call the endpoint with slave_of. What is the policy for REST API
backward compatibility?

I guess it depends on the Trove team. In Nova, backward compatibility in
the API is serious business and that's why we have left all sorts of warts
in the v2.0 API and couldn't remove them. But with the v2.1 API and
microversions, we're able to move the API forward (Ironic also has
microversions, and I think cinder/neutron/keystone are working on adding
that support).

So if maintaining backward compatibility in the trove API is important to
the trove team and it's users, then yes the API change should probably be
reverted. For anyone doing CD with Trove, they are already broken, but
people upgrading from liberty to mitaka could save themselves the break.

(2) Wouldn't [4] just block 2.1.0 in global-requirements on master
(mitaka)?

I'm not sure I understand, [4] here bumps the minimum required version of
python-troveclient to be 2.1.0, not block it. As noted in the review, I
don't know that it's really necessary to bump to 2.1.0 iff we land and
release [1].

(3) Something, potentially your patch set [2], should also fix the
test that is invoking create() with slave_of, no?

[2] is stable/liberty which still has slave_of in the create API, so I
don't think that needs to be fixed in stable/liberty.

-amrith

[1] https://review.openstack.org/290048/
[2] https://review.openstack.org/290023/
[3] https://review.openstack.org/290021/
[4] https://review.openstack.org/290168/
[5] https://review.openstack.org/#/c/245845/
[6] https://review.openstack.org/#/c/245738/
[7] https://review.openstack.org/#/c/246735/

On 03/08/2016 08:24 PM, Matt Riedemann wrote:

On 3/8/2016 4:44 PM, Matt Riedemann wrote:

On 3/8/2016 3:17 PM, Matt Riedemann wrote:

On 3/8/2016 1:18 PM, Amrith Kumar wrote:

Matt,

See inline below.

-amrith

-----Original Message-----
From: Matt Riedemann [mailto:mriedem@linux.vnet.ibm.com]
Sent: Tuesday, March 08, 2016 2:00 PM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [trove][stable] proboscis tests
randomly failing in stable branches; bug 1538506

On 3/8/2016 12:52 PM, Matt Riedemann wrote:

On 3/8/2016 12:35 PM, Amrith Kumar wrote:

Matt,

The correct solution for liberty is that we should fix the tests.
Here's why I believe that this is the case.

In pertinent part, the backtrace from your bug includes:

2016-01-27 07:02:06.777 | File
"/home/jenkins/workspace/periodic-trove-python27-liberty/trove/
tests/

api/replication.py",
line 83, in createslave
2016-01-27 07:02:06.777 | slave
of=instance_info.id)

This is in the tests, not the client.

The test is now generating a parameter that the client no
longer understands.

For a user, here are the various situations that could occur.

  1. User running python-troveclient from the latest 2.1.0 and a
    server from liberty. All is good.
  2. User running an old python-troveclient and a server from
    liberty.
    All is good.

Will this work with python-troveclient 1.2.0 which is the
minimum required troveclient for stable/liberty?

https://github.com/openstack/requirements/blob/stable/liberty/gl
obal-r

equirements.txt#L174

  1. User running an old python-troveclient and a new server from
    mitaka, this is not supported.

How is this not supported? If it's not supported, the minimum
required version of troveclient in master g-r is wrong, since it
hasn't changed since liberty:

https://github.com/openstack/requirements/blob/master/global-req
uireme

nts.txt#L202

I've confirmed that running master (mitaka) unit tests for trove
against python-troveclient 1.2.0 don't work:

http://paste.openstack.org/show/489719/

[amrith] Just like the bug you listed, this appears to be a case
of a broken test. Would you please LP it.

  1. User running a current python-troveclient from the latest
    2.1.0 and a server from mitaka, All is good.

What you have here is a case where a test is generating an
argument
(slave_of) that the client (master, 2.1.0) no longer understands.
There's nothing amiss there, except that the test needs to be
fixed.

When you get back, let's continue the conversation either in
email or IRC #openstack-dev. Looking forward to hearing your
feedback on this.

Thanks,

-amrith

-----Original Message-----
From: Matt Riedemann [mailto:mriedem@linux.vnet.ibm.com]
Sent: Tuesday, March 08, 2016 12:11 PM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [trove][stable] proboscis tests
randomly failing in stable branches; bug 1538506

On 3/8/2016 10:17 AM, Matt Riedemann wrote:

This is a call for help on resolving bug 1538506 [1] where
the proboscis tests randomly fail on the stable branches with
something
like:

TypeError: create() got an unexpected keyword argument
'slave_of'

Craig Vyvial has a proposed stable/kilo change [2] but it has
some issues, at least from me as a reviewer of that change.

The tests are failing the periodic-stable job daily [3].

Can we get someone to help out with this? Otherwise I'm
inclined to say the tests should be disabled on the stable
branches, but that's pretty drastic.

Note that this is the kind of thing that breaks stable branch
policy, which is going to be part of a review when applying
for the stable:follows-policy tag. [4] And the
stable:follows-policy tag may be used to determine when a
stable branch for a project goes end of life if it's not
being maintained.

[1] https://bugs.launchpad.net/trove/+bug/1538506
[2] https://review.openstack.org/#/c/276934/
[3] http://goo.gl/fqf11U
[4]
http://governance.openstack.org/reference/tags/stable_follows
-polic

y.h
tml

This is my solution for liberty, cap python-troveclient<2.1.0
in
global- requirements on stable/liberty [1].

Then there is a change to trove on stable/liberty to use the
g-r version range for python-troveclient for unit tests [2].

Alternatively, we could avoid the cap in stable/liberty by:

  1. Reverting https://review.openstack.org/#/c/245738/
  2. Blacklisting python-troveclient 2.1.0 in g-r 3. Release
    python- troveclient 2.2.0.

It's getting late in the mitaka release to be messing with
this though since we're already past client freeze.

[1] https://review.openstack.org/#/c/290021/
[2] https://review.openstack.org/#/c/290023/

--

Thanks,

Matt Riedemann




OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-
dev




OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-d
ev

--

Thanks,

Matt Riedemann



OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Well, the point I'm trying to get to is, I don't think troveclient
<
2.1.0 will work with the trove-api in mitaka.

For example, in my change to revert the slave_of removal in the
client:

https://review.openstack.org/#/c/290048/

That fails in the API:

http://logs.openstack.org/48/290048/2/check/gate-trove-functional-d
svm-mysql/ddd3089/logs/screen-tr-api.txt.gz#_2016-03-08_19_30_32_79
0

"Validation error: instance Additional properties are not allowed
(u'slave_of' was unexpected)"

So if my client application that I wrote in liberty was using
slave_of, it's happily working until the cloud that my application
is running on upgrades to mitaka, then things break.

Part of the issue here is the backward incompatible change in the
trove API. The other part is, the trove API in mitaka requires
troveclient>=2.1.0 because that's what removed slave_of.

Right?

I'm trying to sort out what a valid troveclient is for master,
because I don't really trust what's in global-requirements since
trove hasn't been testing with those versions, at least not at the
lower bound of 1.2.0.

Here is a change that bumps the minimum required version of
python-troveclient for mitaka to 2.1.0:

https://review.openstack.org/#/c/290168/

Here is my preferred option now.

  1. Revert the slave_of removal from python-troveclient and provide it
    as a compat shim until liberty-eol:

https://review.openstack.org/#/c/290048/

If slaveof is used, a deprecation warning is emitted (which we
should have been doing when this was deprecated in the first place).
It also never sends slave
of to the API, it's just a proxy to
replica_of. So this should work with both the liberty and mitaka
versions of the trove API.

  1. Block python-troveclient 2.1.0 from stable/liberty g-r:

https://review.openstack.org/#/c/290021/

That blocks the bad 2.1.0 version from being used in stable/liberty
which is needed for...

  1. Using the g-r versions of python-troveclient when testing on
    stable/liberty:

https://review.openstack.org/#/c/290023/

If trove is following global-requirements, it should be using the
versions of python-troveclient from global-requirements rather than
installing it from the latest tarball on trunk. master and
stable/kilo are already doing this.


____ OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--

Thanks,

Matt Riedemann


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Mar 9, 2016 in openstack-dev by Amrith_Kumar (10,500 points)   2 4 4

4 Responses

0 votes

On 3/9/2016 1:34 PM, Amrith Kumar wrote:
Matt,

We discussed this at the Trove meeting. Here's my current understanding of the situation:

  1. Your change https://review.openstack.org/#/c/290048/ which reverts the trove client side of the change should be merged.

  2. This change (the Trove API side) https://review.openstack.org/#/c/245845/ should also be reverted. I'm assuming that you'll submit a change for this as well for completeness.

I can submit a change for this.

  1. We need to blacklist python-troveclient 2.1.0 on Liberty, this is your change https://review.openstack.org/#/c/290021/

  2. We need to blacklist python-troveclient 2.1.0 on master, this is currently NOT what your change https://review.openstack.org/290168 does. I disagree with the approach of just increasing the minimum requirement from 1.2.0 to >=2.1.0.

I'm OK with blacklisting 2.1.0 on master and can make that change in the
same patch.

  1. We need to request a new python-troveclient. Whether it should be called 2.1.1 or 2.2.0, I'm not positive. My thought is 2.1.1. But out of an abundance of caution, I'm going to look to #openstack-release/#openstack-infra for guidance on this.

There are no other changes to python-troveclient since 2.1.0 was released:

user@xubuntu:~/git/python-troveclient$ git log --oneline --no-merges 2.1.0..
user@xubuntu:~/git/python-troveclient$

So 2.1.1 is fine.

Thanks,

-amrith

-----Original Message-----
From: Matt Riedemann [mailto:mriedem@linux.vnet.ibm.com]
Sent: Wednesday, March 09, 2016 12:42 PM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [trove][stable] proboscis tests randomly
failing in stable branches; bug 1538506

On 3/9/2016 8:54 AM, Amrith Kumar wrote:

Matt,

As I understand it, you have 4 changes up for review.

  Change [1] seeks to revert the change [6] to python-troveclient on
  master and reinstate the slave_of parameter.

  Change [2] is doing for stable/liberty, the same things that were
  done for master in [9] on master, and [7] on Kilo earlier.

  Change [3] is looking to blacklist python-troveclient 2.1.0 on
  stable/liberty.

  Change [4] is looking to bump the minimum python-troveclient

version
on master to 2.1.0.

Right, and there was actually another for that beat me to this:

https://review.openstack.org/#/c/290115/

Here are three questions I have:

(1) Wouldn't backward compatibility also require that you revert the
other change that removed slaveof on the server [5] ? After all, just
like a python client that calls create() with slave
of, a REST client
could call the endpoint with slave_of. What is the policy for REST API
backward compatibility?

I guess it depends on the Trove team. In Nova, backward compatibility in
the API is serious business and that's why we have left all sorts of warts
in the v2.0 API and couldn't remove them. But with the v2.1 API and
microversions, we're able to move the API forward (Ironic also has
microversions, and I think cinder/neutron/keystone are working on adding
that support).

So if maintaining backward compatibility in the trove API is important to
the trove team and it's users, then yes the API change should probably be
reverted. For anyone doing CD with Trove, they are already broken, but
people upgrading from liberty to mitaka could save themselves the break.

(2) Wouldn't [4] just block 2.1.0 in global-requirements on master
(mitaka)?

I'm not sure I understand, [4] here bumps the minimum required version of
python-troveclient to be 2.1.0, not block it. As noted in the review, I
don't know that it's really necessary to bump to 2.1.0 iff we land and
release [1].

(3) Something, potentially your patch set [2], should also fix the
test that is invoking create() with slave_of, no?

[2] is stable/liberty which still has slave_of in the create API, so I
don't think that needs to be fixed in stable/liberty.

-amrith

[1] https://review.openstack.org/290048/
[2] https://review.openstack.org/290023/
[3] https://review.openstack.org/290021/
[4] https://review.openstack.org/290168/
[5] https://review.openstack.org/#/c/245845/
[6] https://review.openstack.org/#/c/245738/
[7] https://review.openstack.org/#/c/246735/

On 03/08/2016 08:24 PM, Matt Riedemann wrote:

On 3/8/2016 4:44 PM, Matt Riedemann wrote:

On 3/8/2016 3:17 PM, Matt Riedemann wrote:

On 3/8/2016 1:18 PM, Amrith Kumar wrote:

Matt,

See inline below.

-amrith

-----Original Message-----
From: Matt Riedemann [mailto:mriedem@linux.vnet.ibm.com]
Sent: Tuesday, March 08, 2016 2:00 PM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [trove][stable] proboscis tests
randomly failing in stable branches; bug 1538506

On 3/8/2016 12:52 PM, Matt Riedemann wrote:

On 3/8/2016 12:35 PM, Amrith Kumar wrote:

Matt,

The correct solution for liberty is that we should fix the tests.
Here's why I believe that this is the case.

In pertinent part, the backtrace from your bug includes:

2016-01-27 07:02:06.777 | File
"/home/jenkins/workspace/periodic-trove-python27-liberty/trove/
tests/

api/replication.py",
line 83, in createslave
2016-01-27 07:02:06.777 | slave
of=instance_info.id)

This is in the tests, not the client.

The test is now generating a parameter that the client no
longer understands.

For a user, here are the various situations that could occur.

  1. User running python-troveclient from the latest 2.1.0 and a
    server from liberty. All is good.
  2. User running an old python-troveclient and a server from
    liberty.
    All is good.

Will this work with python-troveclient 1.2.0 which is the
minimum required troveclient for stable/liberty?

https://github.com/openstack/requirements/blob/stable/liberty/gl
obal-r

equirements.txt#L174

  1. User running an old python-troveclient and a new server from
    mitaka, this is not supported.

How is this not supported? If it's not supported, the minimum
required version of troveclient in master g-r is wrong, since it
hasn't changed since liberty:

https://github.com/openstack/requirements/blob/master/global-req
uireme

nts.txt#L202

I've confirmed that running master (mitaka) unit tests for trove
against python-troveclient 1.2.0 don't work:

http://paste.openstack.org/show/489719/

[amrith] Just like the bug you listed, this appears to be a case
of a broken test. Would you please LP it.

  1. User running a current python-troveclient from the latest
    2.1.0 and a server from mitaka, All is good.

What you have here is a case where a test is generating an
argument
(slave_of) that the client (master, 2.1.0) no longer understands.
There's nothing amiss there, except that the test needs to be
fixed.

When you get back, let's continue the conversation either in
email or IRC #openstack-dev. Looking forward to hearing your
feedback on this.

Thanks,

-amrith

-----Original Message-----
From: Matt Riedemann [mailto:mriedem@linux.vnet.ibm.com]
Sent: Tuesday, March 08, 2016 12:11 PM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [trove][stable] proboscis tests
randomly failing in stable branches; bug 1538506

On 3/8/2016 10:17 AM, Matt Riedemann wrote:

This is a call for help on resolving bug 1538506 [1] where
the proboscis tests randomly fail on the stable branches with
something
like:

TypeError: create() got an unexpected keyword argument
'slave_of'

Craig Vyvial has a proposed stable/kilo change [2] but it has
some issues, at least from me as a reviewer of that change.

The tests are failing the periodic-stable job daily [3].

Can we get someone to help out with this? Otherwise I'm
inclined to say the tests should be disabled on the stable
branches, but that's pretty drastic.

Note that this is the kind of thing that breaks stable branch
policy, which is going to be part of a review when applying
for the stable:follows-policy tag. [4] And the
stable:follows-policy tag may be used to determine when a
stable branch for a project goes end of life if it's not
being maintained.

[1] https://bugs.launchpad.net/trove/+bug/1538506
[2] https://review.openstack.org/#/c/276934/
[3] http://goo.gl/fqf11U
[4]
http://governance.openstack.org/reference/tags/stable_follows
-polic

y.h
tml

This is my solution for liberty, cap python-troveclient<2.1.0
in
global- requirements on stable/liberty [1].

Then there is a change to trove on stable/liberty to use the
g-r version range for python-troveclient for unit tests [2].

Alternatively, we could avoid the cap in stable/liberty by:

  1. Reverting https://review.openstack.org/#/c/245738/
  2. Blacklisting python-troveclient 2.1.0 in g-r 3. Release
    python- troveclient 2.2.0.

It's getting late in the mitaka release to be messing with
this though since we're already past client freeze.

[1] https://review.openstack.org/#/c/290021/
[2] https://review.openstack.org/#/c/290023/

--

Thanks,

Matt Riedemann




OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-
dev




OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-d
ev

--

Thanks,

Matt Riedemann



OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Well, the point I'm trying to get to is, I don't think troveclient
<
2.1.0 will work with the trove-api in mitaka.

For example, in my change to revert the slave_of removal in the
client:

https://review.openstack.org/#/c/290048/

That fails in the API:

http://logs.openstack.org/48/290048/2/check/gate-trove-functional-d
svm-mysql/ddd3089/logs/screen-tr-api.txt.gz#_2016-03-08_19_30_32_79
0

"Validation error: instance Additional properties are not allowed
(u'slave_of' was unexpected)"

So if my client application that I wrote in liberty was using
slave_of, it's happily working until the cloud that my application
is running on upgrades to mitaka, then things break.

Part of the issue here is the backward incompatible change in the
trove API. The other part is, the trove API in mitaka requires
troveclient>=2.1.0 because that's what removed slave_of.

Right?

I'm trying to sort out what a valid troveclient is for master,
because I don't really trust what's in global-requirements since
trove hasn't been testing with those versions, at least not at the
lower bound of 1.2.0.

Here is a change that bumps the minimum required version of
python-troveclient for mitaka to 2.1.0:

https://review.openstack.org/#/c/290168/

Here is my preferred option now.

  1. Revert the slave_of removal from python-troveclient and provide it
    as a compat shim until liberty-eol:

https://review.openstack.org/#/c/290048/

If slaveof is used, a deprecation warning is emitted (which we
should have been doing when this was deprecated in the first place).
It also never sends slave
of to the API, it's just a proxy to
replica_of. So this should work with both the liberty and mitaka
versions of the trove API.

  1. Block python-troveclient 2.1.0 from stable/liberty g-r:

https://review.openstack.org/#/c/290021/

That blocks the bad 2.1.0 version from being used in stable/liberty
which is needed for...

  1. Using the g-r versions of python-troveclient when testing on
    stable/liberty:

https://review.openstack.org/#/c/290023/

If trove is following global-requirements, it should be using the
versions of python-troveclient from global-requirements rather than
installing it from the latest tarball on trunk. master and
stable/kilo are already doing this.


____ OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--

Thanks,

Matt Riedemann


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--

Thanks,

Matt Riedemann


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 9, 2016 by Matt_Riedemann (48,320 points)   3 7 20
0 votes

Thanks Matt.

Agreed on 1, 2, 3, and 4. The reason for #5 is that there are other changes (for which we have FFE's) which will be merged and a new version of the client requested.

Specifically, this change

https://review.openstack.org/290177

Thanks,

-amrith

-----Original Message-----
From: Matt Riedemann [mailto:mriedem@linux.vnet.ibm.com]
Sent: Wednesday, March 09, 2016 2:53 PM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [tc] [trove][stable] proboscis tests randomly
failing in stable branches; bug 1538506

On 3/9/2016 1:34 PM, Amrith Kumar wrote:

Matt,

We discussed this at the Trove meeting. Here's my current understanding
of the situation:

  1. Your change https://review.openstack.org/#/c/290048/ which reverts
    the trove client side of the change should be merged.

  2. This change (the Trove API side)
    https://review.openstack.org/#/c/245845/ should also be reverted. I'm
    assuming that you'll submit a change for this as well for completeness.

I can submit a change for this.

  1. We need to blacklist python-troveclient 2.1.0 on Liberty, this is
    your change https://review.openstack.org/#/c/290021/

  2. We need to blacklist python-troveclient 2.1.0 on master, this is
    currently NOT what your change https://review.openstack.org/290168 does.
    I disagree with the approach of just increasing the minimum requirement
    from 1.2.0 to >=2.1.0.

I'm OK with blacklisting 2.1.0 on master and can make that change in the
same patch.

  1. We need to request a new python-troveclient. Whether it should be
    called 2.1.1 or 2.2.0, I'm not positive. My thought is 2.1.1. But out of
    an abundance of caution, I'm going to look to #openstack-
    release/#openstack-infra for guidance on this.

There are no other changes to python-troveclient since 2.1.0 was released:

user@xubuntu:~/git/python-troveclient$ git log --oneline --no-merges
2.1.0..
user@xubuntu:~/git/python-troveclient$

So 2.1.1 is fine.

Thanks,

-amrith

-----Original Message-----
From: Matt Riedemann [mailto:mriedem@linux.vnet.ibm.com]
Sent: Wednesday, March 09, 2016 12:42 PM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [trove][stable] proboscis tests randomly
failing in stable branches; bug 1538506

On 3/9/2016 8:54 AM, Amrith Kumar wrote:

Matt,

As I understand it, you have 4 changes up for review.

  Change [1] seeks to revert the change [6] to python-troveclient

on
master and reinstate the slave_of parameter.

  Change [2] is doing for stable/liberty, the same things that

were
done for master in [9] on master, and [7] on Kilo earlier.

  Change [3] is looking to blacklist python-troveclient 2.1.0 on
  stable/liberty.

  Change [4] is looking to bump the minimum python-troveclient

version
on master to 2.1.0.

Right, and there was actually another for that beat me to this:

https://review.openstack.org/#/c/290115/

Here are three questions I have:

(1) Wouldn't backward compatibility also require that you revert the
other change that removed slaveof on the server [5] ? After all,
just like a python client that calls create() with slave
of, a REST
client could call the endpoint with slave_of. What is the policy for
REST API backward compatibility?

I guess it depends on the Trove team. In Nova, backward compatibility
in the API is serious business and that's why we have left all sorts
of warts in the v2.0 API and couldn't remove them. But with the v2.1
API and microversions, we're able to move the API forward (Ironic
also has microversions, and I think cinder/neutron/keystone are
working on adding that support).

So if maintaining backward compatibility in the trove API is
important to the trove team and it's users, then yes the API change
should probably be reverted. For anyone doing CD with Trove, they are
already broken, but people upgrading from liberty to mitaka could save
themselves the break.

(2) Wouldn't [4] just block 2.1.0 in global-requirements on master
(mitaka)?

I'm not sure I understand, [4] here bumps the minimum required
version of python-troveclient to be 2.1.0, not block it. As noted in
the review, I don't know that it's really necessary to bump to 2.1.0
iff we land and release [1].

(3) Something, potentially your patch set [2], should also fix the
test that is invoking create() with slave_of, no?

[2] is stable/liberty which still has slave_of in the create API, so
I don't think that needs to be fixed in stable/liberty.

-amrith

[1] https://review.openstack.org/290048/
[2] https://review.openstack.org/290023/
[3] https://review.openstack.org/290021/
[4] https://review.openstack.org/290168/
[5] https://review.openstack.org/#/c/245845/
[6] https://review.openstack.org/#/c/245738/
[7] https://review.openstack.org/#/c/246735/

On 03/08/2016 08:24 PM, Matt Riedemann wrote:

On 3/8/2016 4:44 PM, Matt Riedemann wrote:

On 3/8/2016 3:17 PM, Matt Riedemann wrote:

On 3/8/2016 1:18 PM, Amrith Kumar wrote:

Matt,

See inline below.

-amrith

-----Original Message-----
From: Matt Riedemann [mailto:mriedem@linux.vnet.ibm.com]
Sent: Tuesday, March 08, 2016 2:00 PM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [trove][stable] proboscis tests
randomly failing in stable branches; bug 1538506

On 3/8/2016 12:52 PM, Matt Riedemann wrote:

On 3/8/2016 12:35 PM, Amrith Kumar wrote:

Matt,

The correct solution for liberty is that we should fix the
tests.
Here's why I believe that this is the case.

In pertinent part, the backtrace from your bug includes:

2016-01-27 07:02:06.777 | File
"/home/jenkins/workspace/periodic-trove-python27-liberty/trov
e/
tests/

api/replication.py",
line 83, in createslave
2016-01-27 07:02:06.777 | slave
of=instance_info.id)

This is in the tests, not the client.

The test is now generating a parameter that the client no
longer understands.

For a user, here are the various situations that could occur.

  1. User running python-troveclient from the latest 2.1.0 and
    a server from liberty. All is good.
  2. User running an old python-troveclient and a server from
    liberty.
    All is good.

Will this work with python-troveclient 1.2.0 which is the
minimum required troveclient for stable/liberty?

https://github.com/openstack/requirements/blob/stable/liberty/
gl
obal-r

equirements.txt#L174

  1. User running an old python-troveclient and a new server
    from mitaka, this is not supported.

How is this not supported? If it's not supported, the minimum
required version of troveclient in master g-r is wrong, since
it hasn't changed since liberty:

https://github.com/openstack/requirements/blob/master/global-r
eq
uireme

nts.txt#L202

I've confirmed that running master (mitaka) unit tests for
trove against python-troveclient 1.2.0 don't work:

http://paste.openstack.org/show/489719/

[amrith] Just like the bug you listed, this appears to be a case
of a broken test. Would you please LP it.

  1. User running a current python-troveclient from the latest
    2.1.0 and a server from mitaka, All is good.

What you have here is a case where a test is generating an
argument
(slave_of) that the client (master, 2.1.0) no longer
understands.
There's nothing amiss there, except that the test needs to be
fixed.

When you get back, let's continue the conversation either in
email or IRC #openstack-dev. Looking forward to hearing your
feedback on this.

Thanks,

-amrith

-----Original Message-----
From: Matt Riedemann [mailto:mriedem@linux.vnet.ibm.com]
Sent: Tuesday, March 08, 2016 12:11 PM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [trove][stable] proboscis tests
randomly failing in stable branches; bug 1538506

On 3/8/2016 10:17 AM, Matt Riedemann wrote:

This is a call for help on resolving bug 1538506 [1] where
the proboscis tests randomly fail on the stable branches
with something
like:

TypeError: create() got an unexpected keyword argument
'slave_of'

Craig Vyvial has a proposed stable/kilo change [2] but it
has some issues, at least from me as a reviewer of that
change.

The tests are failing the periodic-stable job daily [3].

Can we get someone to help out with this? Otherwise I'm
inclined to say the tests should be disabled on the stable
branches, but that's pretty drastic.

Note that this is the kind of thing that breaks stable
branch policy, which is going to be part of a review when
applying for the stable:follows-policy tag. [4] And the
stable:follows-policy tag may be used to determine when a
stable branch for a project goes end of life if it's not
being maintained.

[1] https://bugs.launchpad.net/trove/+bug/1538506
[2] https://review.openstack.org/#/c/276934/
[3] http://goo.gl/fqf11U
[4]
http://governance.openstack.org/reference/tags/stable_follo
ws
-polic

y.h
tml

This is my solution for liberty, cap
python-troveclient<2.1.0 in
global- requirements on stable/liberty [1].

Then there is a change to trove on stable/liberty to use the
g-r version range for python-troveclient for unit tests [2].

Alternatively, we could avoid the cap in stable/liberty by:

  1. Reverting https://review.openstack.org/#/c/245738/
  2. Blacklisting python-troveclient 2.1.0 in g-r 3. Release
    python- troveclient 2.2.0.

It's getting late in the mitaka release to be messing with
this though since we're already past client freeze.

[1] https://review.openstack.org/#/c/290021/
[2] https://review.openstack.org/#/c/290023/

--

Thanks,

Matt Riedemann


__


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscrib
e
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstac
k-
dev


__


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
-d
ev

--

Thanks,

Matt Riedemann


__

OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-d
ev


__

OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-de
v

Well, the point I'm trying to get to is, I don't think
troveclient <
2.1.0 will work with the trove-api in mitaka.

For example, in my change to revert the slave_of removal in the
client:

https://review.openstack.org/#/c/290048/

That fails in the API:

http://logs.openstack.org/48/290048/2/check/gate-trove-functional
-d
svm-mysql/ddd3089/logs/screen-tr-api.txt.gz#_2016-03-08_19_30_32_
79
0

"Validation error: instance Additional properties are not allowed
(u'slave_of' was unexpected)"

So if my client application that I wrote in liberty was using
slave_of, it's happily working until the cloud that my
application is running on upgrades to mitaka, then things break.

Part of the issue here is the backward incompatible change in the
trove API. The other part is, the trove API in mitaka requires
troveclient>=2.1.0 because that's what removed slave_of.

Right?

I'm trying to sort out what a valid troveclient is for master,
because I don't really trust what's in global-requirements since
trove hasn't been testing with those versions, at least not at
the lower bound of 1.2.0.

Here is a change that bumps the minimum required version of
python-troveclient for mitaka to 2.1.0:

https://review.openstack.org/#/c/290168/

Here is my preferred option now.

  1. Revert the slave_of removal from python-troveclient and provide
    it as a compat shim until liberty-eol:

https://review.openstack.org/#/c/290048/

If slaveof is used, a deprecation warning is emitted (which we
should have been doing when this was deprecated in the first place).
It also never sends slave
of to the API, it's just a proxy to
replica_of. So this should work with both the liberty and mitaka
versions of the trove API.

  1. Block python-troveclient 2.1.0 from stable/liberty g-r:

https://review.openstack.org/#/c/290021/

That blocks the bad 2.1.0 version from being used in stable/liberty
which is needed for...

  1. Using the g-r versions of python-troveclient when testing on
    stable/liberty:

https://review.openstack.org/#/c/290023/

If trove is following global-requirements, it should be using the
versions of python-troveclient from global-requirements rather than
installing it from the latest tarball on trunk. master and
stable/kilo are already doing this.


__ ____ OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--

Thanks,

Matt Riedemann


_____ OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


____ OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--

Thanks,

Matt Riedemann


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 9, 2016 by Amrith_Kumar (10,500 points)   2 4 4
0 votes

On 3/9/2016 2:15 PM, Amrith Kumar wrote:
Thanks Matt.

Agreed on 1, 2, 3, and 4. The reason for #5 is that there are other changes (for which we have FFE's) which will be merged and a new version of the client requested.

Specifically, this change

https://review.openstack.org/290177

Thanks,

-amrith

-----Original Message-----
From: Matt Riedemann [mailto:mriedem@linux.vnet.ibm.com]
Sent: Wednesday, March 09, 2016 2:53 PM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [tc] [trove][stable] proboscis tests randomly
failing in stable branches; bug 1538506

On 3/9/2016 1:34 PM, Amrith Kumar wrote:

Matt,

We discussed this at the Trove meeting. Here's my current understanding
of the situation:

  1. Your change https://review.openstack.org/#/c/290048/ which reverts
    the trove client side of the change should be merged.

  2. This change (the Trove API side)
    https://review.openstack.org/#/c/245845/ should also be reverted. I'm
    assuming that you'll submit a change for this as well for completeness.

I can submit a change for this.

  1. We need to blacklist python-troveclient 2.1.0 on Liberty, this is
    your change https://review.openstack.org/#/c/290021/

  2. We need to blacklist python-troveclient 2.1.0 on master, this is
    currently NOT what your change https://review.openstack.org/290168 does.
    I disagree with the approach of just increasing the minimum requirement
    from 1.2.0 to >=2.1.0.

I'm OK with blacklisting 2.1.0 on master and can make that change in the
same patch.

  1. We need to request a new python-troveclient. Whether it should be
    called 2.1.1 or 2.2.0, I'm not positive. My thought is 2.1.1. But out of
    an abundance of caution, I'm going to look to #openstack-
    release/#openstack-infra for guidance on this.

There are no other changes to python-troveclient since 2.1.0 was released:

user@xubuntu:~/git/python-troveclient$ git log --oneline --no-merges
2.1.0..
user@xubuntu:~/git/python-troveclient$

So 2.1.1 is fine.

Thanks,

-amrith

-----Original Message-----
From: Matt Riedemann [mailto:mriedem@linux.vnet.ibm.com]
Sent: Wednesday, March 09, 2016 12:42 PM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [trove][stable] proboscis tests randomly
failing in stable branches; bug 1538506

On 3/9/2016 8:54 AM, Amrith Kumar wrote:

Matt,

As I understand it, you have 4 changes up for review.

   Change [1] seeks to revert the change [6] to python-troveclient

on
master and reinstate the slave_of parameter.

   Change [2] is doing for stable/liberty, the same things that

were
done for master in [9] on master, and [7] on Kilo earlier.

   Change [3] is looking to blacklist python-troveclient 2.1.0 on
   stable/liberty.

   Change [4] is looking to bump the minimum python-troveclient

version
on master to 2.1.0.

Right, and there was actually another for that beat me to this:

https://review.openstack.org/#/c/290115/

Here are three questions I have:

(1) Wouldn't backward compatibility also require that you revert the
other change that removed slaveof on the server [5] ? After all,
just like a python client that calls create() with slave
of, a REST
client could call the endpoint with slave_of. What is the policy for
REST API backward compatibility?

I guess it depends on the Trove team. In Nova, backward compatibility
in the API is serious business and that's why we have left all sorts
of warts in the v2.0 API and couldn't remove them. But with the v2.1
API and microversions, we're able to move the API forward (Ironic
also has microversions, and I think cinder/neutron/keystone are
working on adding that support).

So if maintaining backward compatibility in the trove API is
important to the trove team and it's users, then yes the API change
should probably be reverted. For anyone doing CD with Trove, they are
already broken, but people upgrading from liberty to mitaka could save
themselves the break.

(2) Wouldn't [4] just block 2.1.0 in global-requirements on master
(mitaka)?

I'm not sure I understand, [4] here bumps the minimum required
version of python-troveclient to be 2.1.0, not block it. As noted in
the review, I don't know that it's really necessary to bump to 2.1.0
iff we land and release [1].

(3) Something, potentially your patch set [2], should also fix the
test that is invoking create() with slave_of, no?

[2] is stable/liberty which still has slave_of in the create API, so
I don't think that needs to be fixed in stable/liberty.

-amrith

[1] https://review.openstack.org/290048/
[2] https://review.openstack.org/290023/
[3] https://review.openstack.org/290021/
[4] https://review.openstack.org/290168/
[5] https://review.openstack.org/#/c/245845/
[6] https://review.openstack.org/#/c/245738/
[7] https://review.openstack.org/#/c/246735/

On 03/08/2016 08:24 PM, Matt Riedemann wrote:

On 3/8/2016 4:44 PM, Matt Riedemann wrote:

On 3/8/2016 3:17 PM, Matt Riedemann wrote:

On 3/8/2016 1:18 PM, Amrith Kumar wrote:

Matt,

See inline below.

-amrith

-----Original Message-----
From: Matt Riedemann [mailto:mriedem@linux.vnet.ibm.com]
Sent: Tuesday, March 08, 2016 2:00 PM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [trove][stable] proboscis tests
randomly failing in stable branches; bug 1538506

On 3/8/2016 12:52 PM, Matt Riedemann wrote:

On 3/8/2016 12:35 PM, Amrith Kumar wrote:

Matt,

The correct solution for liberty is that we should fix the
tests.
Here's why I believe that this is the case.

In pertinent part, the backtrace from your bug includes:

2016-01-27 07:02:06.777 | File
"/home/jenkins/workspace/periodic-trove-python27-liberty/trov
e/
tests/

api/replication.py",
line 83, in createslave
2016-01-27 07:02:06.777 | slave
of=instance_info.id)

This is in the tests, not the client.

The test is now generating a parameter that the client no
longer understands.

For a user, here are the various situations that could occur.

  1. User running python-troveclient from the latest 2.1.0 and
    a server from liberty. All is good.
  2. User running an old python-troveclient and a server from
    liberty.
    All is good.

Will this work with python-troveclient 1.2.0 which is the
minimum required troveclient for stable/liberty?

https://github.com/openstack/requirements/blob/stable/liberty/
gl
obal-r

equirements.txt#L174

  1. User running an old python-troveclient and a new server
    from mitaka, this is not supported.

How is this not supported? If it's not supported, the minimum
required version of troveclient in master g-r is wrong, since
it hasn't changed since liberty:

https://github.com/openstack/requirements/blob/master/global-r
eq
uireme

nts.txt#L202

I've confirmed that running master (mitaka) unit tests for
trove against python-troveclient 1.2.0 don't work:

http://paste.openstack.org/show/489719/

[amrith] Just like the bug you listed, this appears to be a case
of a broken test. Would you please LP it.

  1. User running a current python-troveclient from the latest
    2.1.0 and a server from mitaka, All is good.

What you have here is a case where a test is generating an
argument
(slave_of) that the client (master, 2.1.0) no longer
understands.
There's nothing amiss there, except that the test needs to be
fixed.

When you get back, let's continue the conversation either in
email or IRC #openstack-dev. Looking forward to hearing your
feedback on this.

Thanks,

-amrith

-----Original Message-----
From: Matt Riedemann [mailto:mriedem@linux.vnet.ibm.com]
Sent: Tuesday, March 08, 2016 12:11 PM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [trove][stable] proboscis tests
randomly failing in stable branches; bug 1538506

On 3/8/2016 10:17 AM, Matt Riedemann wrote:

This is a call for help on resolving bug 1538506 [1] where
the proboscis tests randomly fail on the stable branches
with something
like:

TypeError: create() got an unexpected keyword argument
'slave_of'

Craig Vyvial has a proposed stable/kilo change [2] but it
has some issues, at least from me as a reviewer of that
change.

The tests are failing the periodic-stable job daily [3].

Can we get someone to help out with this? Otherwise I'm
inclined to say the tests should be disabled on the stable
branches, but that's pretty drastic.

Note that this is the kind of thing that breaks stable
branch policy, which is going to be part of a review when
applying for the stable:follows-policy tag. [4] And the
stable:follows-policy tag may be used to determine when a
stable branch for a project goes end of life if it's not
being maintained.

[1] https://bugs.launchpad.net/trove/+bug/1538506
[2] https://review.openstack.org/#/c/276934/
[3] http://goo.gl/fqf11U
[4]
http://governance.openstack.org/reference/tags/stable_follo
ws
-polic

y.h
tml

This is my solution for liberty, cap
python-troveclient<2.1.0 in
global- requirements on stable/liberty [1].

Then there is a change to trove on stable/liberty to use the
g-r version range for python-troveclient for unit tests [2].

Alternatively, we could avoid the cap in stable/liberty by:

  1. Reverting https://review.openstack.org/#/c/245738/
  2. Blacklisting python-troveclient 2.1.0 in g-r 3. Release
    python- troveclient 2.2.0.

It's getting late in the mitaka release to be messing with
this though since we're already past client freeze.

[1] https://review.openstack.org/#/c/290021/
[2] https://review.openstack.org/#/c/290023/

--

Thanks,

Matt Riedemann


__


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscrib
e
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstac
k-
dev


__


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
-d
ev

--

Thanks,

Matt Riedemann


__

OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-d
ev


__

OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-de
v

Well, the point I'm trying to get to is, I don't think
troveclient <
2.1.0 will work with the trove-api in mitaka.

For example, in my change to revert the slave_of removal in the
client:

https://review.openstack.org/#/c/290048/

That fails in the API:

http://logs.openstack.org/48/290048/2/check/gate-trove-functional
-d
svm-mysql/ddd3089/logs/screen-tr-api.txt.gz#_2016-03-08_19_30_32_
79
0

"Validation error: instance Additional properties are not allowed
(u'slave_of' was unexpected)"

So if my client application that I wrote in liberty was using
slave_of, it's happily working until the cloud that my
application is running on upgrades to mitaka, then things break.

Part of the issue here is the backward incompatible change in the
trove API. The other part is, the trove API in mitaka requires
troveclient>=2.1.0 because that's what removed slave_of.

Right?

I'm trying to sort out what a valid troveclient is for master,
because I don't really trust what's in global-requirements since
trove hasn't been testing with those versions, at least not at
the lower bound of 1.2.0.

Here is a change that bumps the minimum required version of
python-troveclient for mitaka to 2.1.0:

https://review.openstack.org/#/c/290168/

Here is my preferred option now.

  1. Revert the slave_of removal from python-troveclient and provide
    it as a compat shim until liberty-eol:

https://review.openstack.org/#/c/290048/

If slaveof is used, a deprecation warning is emitted (which we
should have been doing when this was deprecated in the first place).
It also never sends slave
of to the API, it's just a proxy to
replica_of. So this should work with both the liberty and mitaka
versions of the trove API.

  1. Block python-troveclient 2.1.0 from stable/liberty g-r:

https://review.openstack.org/#/c/290021/

That blocks the bad 2.1.0 version from being used in stable/liberty
which is needed for...

  1. Using the g-r versions of python-troveclient when testing on
    stable/liberty:

https://review.openstack.org/#/c/290023/

If trove is following global-requirements, it should be using the
versions of python-troveclient from global-requirements rather than
installing it from the latest tarball on trunk. master and
stable/kilo are already doing this.


__ ____ OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--

Thanks,

Matt Riedemann


_____ OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


____ OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--

Thanks,

Matt Riedemann


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

I'd recommend that you land the revert in the client and release that as
2.1.1, then worry about an FFE for the client since that's going to be a
2.2.0 minor version bump.

--

Thanks,

Matt Riedemann


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 9, 2016 by Matt_Riedemann (48,320 points)   3 7 20
0 votes

On 3/9/2016 3:14 PM, Matt Riedemann wrote:

On 3/9/2016 2:15 PM, Amrith Kumar wrote:

Thanks Matt.

Agreed on 1, 2, 3, and 4. The reason for #5 is that there are other
changes (for which we have FFE's) which will be merged and a new
version of the client requested.

Specifically, this change

https://review.openstack.org/290177

Thanks,

-amrith

-----Original Message-----
From: Matt Riedemann [mailto:mriedem@linux.vnet.ibm.com]
Sent: Wednesday, March 09, 2016 2:53 PM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [tc] [trove][stable] proboscis tests
randomly
failing in stable branches; bug 1538506

On 3/9/2016 1:34 PM, Amrith Kumar wrote:

Matt,

We discussed this at the Trove meeting. Here's my current understanding
of the situation:

  1. Your change https://review.openstack.org/#/c/290048/ which reverts
    the trove client side of the change should be merged.

  2. This change (the Trove API side)
    https://review.openstack.org/#/c/245845/ should also be reverted. I'm
    assuming that you'll submit a change for this as well for completeness.

I can submit a change for this.

  1. We need to blacklist python-troveclient 2.1.0 on Liberty, this is
    your change https://review.openstack.org/#/c/290021/

  2. We need to blacklist python-troveclient 2.1.0 on master, this is
    currently NOT what your change https://review.openstack.org/290168
    does.
    I disagree with the approach of just increasing the minimum requirement
    from 1.2.0 to >=2.1.0.

I'm OK with blacklisting 2.1.0 on master and can make that change in the
same patch.

  1. We need to request a new python-troveclient. Whether it should be
    called 2.1.1 or 2.2.0, I'm not positive. My thought is 2.1.1. But out of
    an abundance of caution, I'm going to look to #openstack-
    release/#openstack-infra for guidance on this.

There are no other changes to python-troveclient since 2.1.0 was
released:

user@xubuntu:~/git/python-troveclient$ git log --oneline --no-merges
2.1.0..
user@xubuntu:~/git/python-troveclient$

So 2.1.1 is fine.

Thanks,

-amrith

-----Original Message-----
From: Matt Riedemann [mailto:mriedem@linux.vnet.ibm.com]
Sent: Wednesday, March 09, 2016 12:42 PM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [trove][stable] proboscis tests randomly
failing in stable branches; bug 1538506

On 3/9/2016 8:54 AM, Amrith Kumar wrote:

Matt,

As I understand it, you have 4 changes up for review.

   Change [1] seeks to revert the change [6] to

python-troveclient
on
master and reinstate the slave_of parameter.

   Change [2] is doing for stable/liberty, the same things that

were
done for master in [9] on master, and [7] on Kilo earlier.

   Change [3] is looking to blacklist python-troveclient 2.1.0 on
   stable/liberty.

   Change [4] is looking to bump the minimum python-troveclient

version
on master to 2.1.0.

Right, and there was actually another for that beat me to this:

https://review.openstack.org/#/c/290115/

Here are three questions I have:

(1) Wouldn't backward compatibility also require that you revert the
other change that removed slaveof on the server [5] ? After all,
just like a python client that calls create() with slave
of, a REST
client could call the endpoint with slave_of. What is the policy for
REST API backward compatibility?

I guess it depends on the Trove team. In Nova, backward compatibility
in the API is serious business and that's why we have left all sorts
of warts in the v2.0 API and couldn't remove them. But with the v2.1
API and microversions, we're able to move the API forward (Ironic
also has microversions, and I think cinder/neutron/keystone are
working on adding that support).

So if maintaining backward compatibility in the trove API is
important to the trove team and it's users, then yes the API change
should probably be reverted. For anyone doing CD with Trove, they are
already broken, but people upgrading from liberty to mitaka could save
themselves the break.

(2) Wouldn't [4] just block 2.1.0 in global-requirements on master
(mitaka)?

I'm not sure I understand, [4] here bumps the minimum required
version of python-troveclient to be 2.1.0, not block it. As noted in
the review, I don't know that it's really necessary to bump to 2.1.0
iff we land and release [1].

(3) Something, potentially your patch set [2], should also fix the
test that is invoking create() with slave_of, no?

[2] is stable/liberty which still has slave_of in the create API, so
I don't think that needs to be fixed in stable/liberty.

-amrith

[1] https://review.openstack.org/290048/
[2] https://review.openstack.org/290023/
[3] https://review.openstack.org/290021/
[4] https://review.openstack.org/290168/
[5] https://review.openstack.org/#/c/245845/
[6] https://review.openstack.org/#/c/245738/
[7] https://review.openstack.org/#/c/246735/

On 03/08/2016 08:24 PM, Matt Riedemann wrote:

On 3/8/2016 4:44 PM, Matt Riedemann wrote:

On 3/8/2016 3:17 PM, Matt Riedemann wrote:

On 3/8/2016 1:18 PM, Amrith Kumar wrote:

Matt,

See inline below.

-amrith

-----Original Message-----
From: Matt Riedemann [mailto:mriedem@linux.vnet.ibm.com]
Sent: Tuesday, March 08, 2016 2:00 PM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [trove][stable] proboscis tests
randomly failing in stable branches; bug 1538506

On 3/8/2016 12:52 PM, Matt Riedemann wrote:

On 3/8/2016 12:35 PM, Amrith Kumar wrote:

Matt,

The correct solution for liberty is that we should fix the
tests.
Here's why I believe that this is the case.

In pertinent part, the backtrace from your bug includes:

2016-01-27 07:02:06.777 | File
"/home/jenkins/workspace/periodic-trove-python27-liberty/trov
e/
tests/

api/replication.py",
line 83, in createslave
2016-01-27 07:02:06.777 | slave
of=instance_info.id)

This is in the tests, not the client.

The test is now generating a parameter that the client no
longer understands.

For a user, here are the various situations that could occur.

  1. User running python-troveclient from the latest 2.1.0 and
    a server from liberty. All is good.
  2. User running an old python-troveclient and a server from
    liberty.
    All is good.

Will this work with python-troveclient 1.2.0 which is the
minimum required troveclient for stable/liberty?

https://github.com/openstack/requirements/blob/stable/liberty/
gl
obal-r

equirements.txt#L174

  1. User running an old python-troveclient and a new server
    from mitaka, this is not supported.

How is this not supported? If it's not supported, the minimum
required version of troveclient in master g-r is wrong, since
it hasn't changed since liberty:

https://github.com/openstack/requirements/blob/master/global-r
eq
uireme

nts.txt#L202

I've confirmed that running master (mitaka) unit tests for
trove against python-troveclient 1.2.0 don't work:

http://paste.openstack.org/show/489719/

[amrith] Just like the bug you listed, this appears to be a case
of a broken test. Would you please LP it.

  1. User running a current python-troveclient from the latest
    2.1.0 and a server from mitaka, All is good.

What you have here is a case where a test is generating an
argument
(slave_of) that the client (master, 2.1.0) no longer
understands.
There's nothing amiss there, except that the test needs to be
fixed.

When you get back, let's continue the conversation either in
email or IRC #openstack-dev. Looking forward to hearing your
feedback on this.

Thanks,

-amrith

-----Original Message-----
From: Matt Riedemann [mailto:mriedem@linux.vnet.ibm.com]
Sent: Tuesday, March 08, 2016 12:11 PM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [trove][stable] proboscis tests
randomly failing in stable branches; bug 1538506

On 3/8/2016 10:17 AM, Matt Riedemann wrote:

This is a call for help on resolving bug 1538506 [1] where
the proboscis tests randomly fail on the stable branches
with something
like:

TypeError: create() got an unexpected keyword argument
'slave_of'

Craig Vyvial has a proposed stable/kilo change [2] but it
has some issues, at least from me as a reviewer of that
change.

The tests are failing the periodic-stable job daily [3].

Can we get someone to help out with this? Otherwise I'm
inclined to say the tests should be disabled on the stable
branches, but that's pretty drastic.

Note that this is the kind of thing that breaks stable
branch policy, which is going to be part of a review when
applying for the stable:follows-policy tag. [4] And the
stable:follows-policy tag may be used to determine when a
stable branch for a project goes end of life if it's not
being maintained.

[1] https://bugs.launchpad.net/trove/+bug/1538506
[2] https://review.openstack.org/#/c/276934/
[3] http://goo.gl/fqf11U
[4]
http://governance.openstack.org/reference/tags/stable_follo
ws
-polic

y.h
tml

This is my solution for liberty, cap
python-troveclient<2.1.0 in
global- requirements on stable/liberty [1].

Then there is a change to trove on stable/liberty to use the
g-r version range for python-troveclient for unit tests [2].

Alternatively, we could avoid the cap in stable/liberty by:

  1. Reverting https://review.openstack.org/#/c/245738/
  2. Blacklisting python-troveclient 2.1.0 in g-r 3. Release
    python- troveclient 2.2.0.

It's getting late in the mitaka release to be messing with
this though since we're already past client freeze.

[1] https://review.openstack.org/#/c/290021/
[2] https://review.openstack.org/#/c/290023/

--

Thanks,

Matt Riedemann


__


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscrib
e
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstac
k-
dev


__


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
-d
ev

--

Thanks,

Matt Riedemann


__

OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-d
ev


__

OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-de
v

Well, the point I'm trying to get to is, I don't think
troveclient <
2.1.0 will work with the trove-api in mitaka.

For example, in my change to revert the slave_of removal in the
client:

https://review.openstack.org/#/c/290048/

That fails in the API:

http://logs.openstack.org/48/290048/2/check/gate-trove-functional
-d
svm-mysql/ddd3089/logs/screen-tr-api.txt.gz#_2016-03-08_19_30_32_
79
0

"Validation error: instance Additional properties are not allowed
(u'slave_of' was unexpected)"

So if my client application that I wrote in liberty was using
slave_of, it's happily working until the cloud that my
application is running on upgrades to mitaka, then things break.

Part of the issue here is the backward incompatible change in the
trove API. The other part is, the trove API in mitaka requires
troveclient>=2.1.0 because that's what removed slave_of.

Right?

I'm trying to sort out what a valid troveclient is for master,
because I don't really trust what's in global-requirements since
trove hasn't been testing with those versions, at least not at
the lower bound of 1.2.0.

Here is a change that bumps the minimum required version of
python-troveclient for mitaka to 2.1.0:

https://review.openstack.org/#/c/290168/

Here is my preferred option now.

  1. Revert the slave_of removal from python-troveclient and provide
    it as a compat shim until liberty-eol:

https://review.openstack.org/#/c/290048/

If slaveof is used, a deprecation warning is emitted (which we
should have been doing when this was deprecated in the first place).
It also never sends slave
of to the API, it's just a proxy to
replica_of. So this should work with both the liberty and mitaka
versions of the trove API.

  1. Block python-troveclient 2.1.0 from stable/liberty g-r:

https://review.openstack.org/#/c/290021/

That blocks the bad 2.1.0 version from being used in stable/liberty
which is needed for...

  1. Using the g-r versions of python-troveclient when testing on
    stable/liberty:

https://review.openstack.org/#/c/290023/

If trove is following global-requirements, it should be using the
versions of python-troveclient from global-requirements rather than
installing it from the latest tarball on trunk. master and
stable/kilo are already doing this.


__ ____ OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--

Thanks,

Matt Riedemann


_____ OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


____ OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--

Thanks,

Matt Riedemann


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

I'd recommend that you land the revert in the client and release that as
2.1.1, then worry about an FFE for the client since that's going to be a
2.2.0 minor version bump.

I think the last problem now is, the reno change for trove on
stable/liberty is failing because the mysql job always fails:

http://logs.openstack.org/84/252584/8/check/gate-trove-functional-dsvm-mysql/072142e/console.html#_2016-03-10_03_02_17_051

Is that a known issue? Was something fixed on master that can be backported?

--

Thanks,

Matt Riedemann


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 10, 2016 by Matt_Riedemann (48,320 points)   3 7 20
...