settingsLogin | Registersettings

[openstack-dev] Let's drop the postgresql gate job

0 votes

It's that time of year again to talk about killing this job, at least
from the integrated gate (move it to experimental for people that care
about postgresql, or make it gating on a smaller subset of projects like
oslo.db).

The postgresql job used to have three interesting things about it:

  1. It ran keystone with eventlet (which is no longer a thing).
  2. It runs the n-api-meta service rather than using config drive.
  3. It uses postgresql for the database.

So #1 is gone, and for #3, according to the April 2016 user survey (page
40) [1], 4% of reporting deployments are using it in production.

I don't think we're running n-api-meta in any other integrated gate
jobs, but I'm pretty sure there is at least one neutron job out there
that's running with it that way. We could also consider making the
nova-net dsvm full gate job run n-api-meta, or vice-versa with the
neutron dsvm full gate job.

We also have to consider that with HP public cloud being gone as a node
provider and we've got fewer test nodes to run with, we have to make
tough decisions about which jobs we're going to run in the integrated gate.

I'm bringing this up again because Nova has a few more jobs it would
like to make voting on it's repo (neutron LB and live migration, at
least in the check queue) but there are concerns about adding yet more
jobs that each change has to get through before it's merged, which means
if anything goes wrong in any of those we can have a 24 hour turnaround
on getting an approved change back through the gate.

[1]
https://www.openstack.org/assets/survey/April-2016-User-Survey-Report.pdf

--

Thanks,

Matt Riedemann


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Aug 18, 2016 in openstack-dev by Matt_Riedemann (48,320 points)   3 7 21
retagged Jan 26, 2017 by admin

14 Responses

0 votes

On 08/18/2016 10:00 AM, Matt Riedemann wrote:
It's that time of year again to talk about killing this job, at least
from the integrated gate (move it to experimental for people that care
about postgresql, or make it gating on a smaller subset of projects like
oslo.db).

The postgresql job used to have three interesting things about it:

  1. It ran keystone with eventlet (which is no longer a thing).
  2. It runs the n-api-meta service rather than using config drive.
  3. It uses postgresql for the database.

So #1 is gone, and for #3, according to the April 2016 user survey (page
40) [1], 4% of reporting deployments are using it in production.

I don't think we're running n-api-meta in any other integrated gate
jobs, but I'm pretty sure there is at least one neutron job out there
that's running with it that way. We could also consider making the
nova-net dsvm full gate job run n-api-meta, or vice-versa with the
neutron dsvm full gate job.

We also have to consider that with HP public cloud being gone as a node
provider and we've got fewer test nodes to run with, we have to make
tough decisions about which jobs we're going to run in the integrated gate.

I'm bringing this up again because Nova has a few more jobs it would
like to make voting on it's repo (neutron LB and live migration, at
least in the check queue) but there are concerns about adding yet more
jobs that each change has to get through before it's merged, which means
if anything goes wrong in any of those we can have a 24 hour turnaround
on getting an approved change back through the gate.

[1]
https://www.openstack.org/assets/survey/April-2016-User-Survey-Report.pdf

I don't know about nova, but at least in keystone when I was testing
upgrades I found an error that had to be fixed before release of Mitaka.
Guess I'm part of the 4% :(

--
-- Matthew Thode (prometheanfire)


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

responded Aug 18, 2016 by prometheanfire_at_ge (6,880 points)   1 4 5
0 votes

On Thu, Aug 18, 2016 at 11:33:59AM -0500, Matthew Thode wrote:
On 08/18/2016 10:00 AM, Matt Riedemann wrote:

It's that time of year again to talk about killing this job, at least
from the integrated gate (move it to experimental for people that care
about postgresql, or make it gating on a smaller subset of projects like
oslo.db).

The postgresql job used to have three interesting things about it:

  1. It ran keystone with eventlet (which is no longer a thing).
  2. It runs the n-api-meta service rather than using config drive.
  3. It uses postgresql for the database.

So #1 is gone, and for #3, according to the April 2016 user survey (page
40) [1], 4% of reporting deployments are using it in production.

I don't think we're running n-api-meta in any other integrated gate
jobs, but I'm pretty sure there is at least one neutron job out there
that's running with it that way. We could also consider making the
nova-net dsvm full gate job run n-api-meta, or vice-versa with the
neutron dsvm full gate job.

We also have to consider that with HP public cloud being gone as a node
provider and we've got fewer test nodes to run with, we have to make
tough decisions about which jobs we're going to run in the integrated gate.

I'm bringing this up again because Nova has a few more jobs it would
like to make voting on it's repo (neutron LB and live migration, at
least in the check queue) but there are concerns about adding yet more
jobs that each change has to get through before it's merged, which means
if anything goes wrong in any of those we can have a 24 hour turnaround
on getting an approved change back through the gate.

[1]
https://www.openstack.org/assets/survey/April-2016-User-Survey-Report.pdf

I don't know about nova, but at least in keystone when I was testing
upgrades I found an error that had to be fixed before release of Mitaka.
Guess I'm part of the 4% :(

That's not what we're talking about here. Your issue most likely stemmed from
keystone's lack of tests that do DB migrations with real data. The proposal here
is not talking about stopping all testing on postgres, just removing the postgres
dsvm tempset jobs from the integrated gate. Those jobs have very limited
additional value for the reasons Matt outlined. They also clearly did not catch
your upgrade issue and most (if not all) of the other postgres issues are caught
with are more targeted testing of the db layer done in the project repos.

-Matt Treinish


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

responded Aug 18, 2016 by Matthew_Treinish (11,200 points)   2 5 5
0 votes

On 08/18/2016 10:00 AM, Matt Riedemann wrote:
It's that time of year again to talk about killing this job, at least
from the integrated gate (move it to experimental for people that care
about postgresql, or make it gating on a smaller subset of projects like
oslo.db).

The postgresql job used to have three interesting things about it:

  1. It ran keystone with eventlet (which is no longer a thing).
  2. It runs the n-api-meta service rather than using config drive.
  3. It uses postgresql for the database.

So #1 is gone, and for #3, according to the April 2016 user survey (page
40) [1], 4% of reporting deployments are using it in production.

I don't think we're running n-api-meta in any other integrated gate
jobs, but I'm pretty sure there is at least one neutron job out there
that's running with it that way. We could also consider making the
nova-net dsvm full gate job run n-api-meta, or vice-versa with the
neutron dsvm full gate job.

We also have to consider that with HP public cloud being gone as a node
provider and we've got fewer test nodes to run with, we have to make
tough decisions about which jobs we're going to run in the integrated gate.

I'm bringing this up again because Nova has a few more jobs it would
like to make voting on it's repo (neutron LB and live migration, at
least in the check queue) but there are concerns about adding yet more
jobs that each change has to get through before it's merged, which means
if anything goes wrong in any of those we can have a 24 hour turnaround
on getting an approved change back through the gate.

[1]
https://www.openstack.org/assets/survey/April-2016-User-Survey-Report.pdf

Perhaps a better option would be to get oslo.db to run cross-project
checks like we do in requirements. That way the right team is covering
the usage of postgres and we still have coverage while still lowering
gate load for most projects.

--
-- Matthew Thode (prometheanfire)


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

responded Aug 18, 2016 by prometheanfire_at_ge (6,880 points)   1 4 5
0 votes

On 08/18/2016 11:00 AM, Matt Riedemann wrote:
It's that time of year again to talk about killing this job, at least
from the integrated gate (move it to experimental for people that care
about postgresql, or make it gating on a smaller subset of projects like
oslo.db).

The postgresql job used to have three interesting things about it:

  1. It ran keystone with eventlet (which is no longer a thing).
  2. It runs the n-api-meta service rather than using config drive.
  3. It uses postgresql for the database.

So #1 is gone, and for #3, according to the April 2016 user survey (page
40) [1], 4% of reporting deployments are using it in production.

I don't think we're running n-api-meta in any other integrated gate
jobs, but I'm pretty sure there is at least one neutron job out there
that's running with it that way. We could also consider making the
nova-net dsvm full gate job run n-api-meta, or vice-versa with the
neutron dsvm full gate job.

We also have to consider that with HP public cloud being gone as a node
provider and we've got fewer test nodes to run with, we have to make
tough decisions about which jobs we're going to run in the integrated gate.

I'm bringing this up again because Nova has a few more jobs it would
like to make voting on it's repo (neutron LB and live migration, at
least in the check queue) but there are concerns about adding yet more
jobs that each change has to get through before it's merged, which means
if anything goes wrong in any of those we can have a 24 hour turnaround
on getting an approved change back through the gate.

[1]
https://www.openstack.org/assets/survey/April-2016-User-Survey-Report.pdf

+1.

Postgresql in the gate hasn't provided any real value in a long time
(tempest just really can't tickle the differences between the dbs,
especially as projects put much better input validation in place).
During icehouse the job was even accidentally running mysql for 6 weeks,
and no one noticed.

-Sean

--
Sean Dague
http://dague.net


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Aug 18, 2016 by Sean_Dague (66,200 points)   4 8 14
0 votes

On 8/18/2016 12:14 PM, Matthew Treinish wrote:
On Thu, Aug 18, 2016 at 11:33:59AM -0500, Matthew Thode wrote:

On 08/18/2016 10:00 AM, Matt Riedemann wrote:

It's that time of year again to talk about killing this job, at least
from the integrated gate (move it to experimental for people that care
about postgresql, or make it gating on a smaller subset of projects like
oslo.db).

The postgresql job used to have three interesting things about it:

  1. It ran keystone with eventlet (which is no longer a thing).
  2. It runs the n-api-meta service rather than using config drive.
  3. It uses postgresql for the database.

So #1 is gone, and for #3, according to the April 2016 user survey (page
40) [1], 4% of reporting deployments are using it in production.

I don't think we're running n-api-meta in any other integrated gate
jobs, but I'm pretty sure there is at least one neutron job out there
that's running with it that way. We could also consider making the
nova-net dsvm full gate job run n-api-meta, or vice-versa with the
neutron dsvm full gate job.

We also have to consider that with HP public cloud being gone as a node
provider and we've got fewer test nodes to run with, we have to make
tough decisions about which jobs we're going to run in the integrated gate.

I'm bringing this up again because Nova has a few more jobs it would
like to make voting on it's repo (neutron LB and live migration, at
least in the check queue) but there are concerns about adding yet more
jobs that each change has to get through before it's merged, which means
if anything goes wrong in any of those we can have a 24 hour turnaround
on getting an approved change back through the gate.

[1]
https://www.openstack.org/assets/survey/April-2016-User-Survey-Report.pdf

I don't know about nova, but at least in keystone when I was testing
upgrades I found an error that had to be fixed before release of Mitaka.
Guess I'm part of the 4% :(

That's not what we're talking about here. Your issue most likely stemmed from
keystone's lack of tests that do DB migrations with real data. The proposal here
is not talking about stopping all testing on postgres, just removing the postgres
dsvm tempset jobs from the integrated gate. Those jobs have very limited
additional value for the reasons Matt outlined. They also clearly did not catch
your upgrade issue and most (if not all) of the other postgres issues are caught
with are more targeted testing of the db layer done in the project repos.

-Matt Treinish


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Good point. Another good point that came up in IRC was that the pg job
made more sense in the integrated gate for testing postgresql as a DB
backend before we had oslo.db in all of the projects. Because back in
the day we were all just getting the DB API code from oslo-incubator and
it had some PG-specific conditionals in it, but that's all abstracted in
oslo.db now which everyone should be using, or at least moving to since
oslo-incubator is EOL.

So definitely gating with a PG backend for oslo.db is a good thing to
still do, it just makes less sense for everything else that's running
with the integrated gate jobs.

Another thing that came up is that even if we start running n-api-meta
in other jobs (or all jobs), Tempest will still test forcing an instance
to boot with config drive here:

https://github.com/openstack/tempest/blob/6b1df9fdf43b298d029e032fe8a737548218c1bf/tempest/scenario/test_server_basic_ops.py#L134

That's config-driven but it's the default in Tempest so it's what we'd
be using in the gate.

--

Thanks,

Matt Riedemann


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Aug 18, 2016 by Matt_Riedemann (48,320 points)   3 7 21
0 votes

Excerpts from Matthew Thode's message of 2016-08-18 13:18:05 -0500:

On 08/18/2016 10:00 AM, Matt Riedemann wrote:

It's that time of year again to talk about killing this job, at least
from the integrated gate (move it to experimental for people that care
about postgresql, or make it gating on a smaller subset of projects like
oslo.db).

The postgresql job used to have three interesting things about it:

  1. It ran keystone with eventlet (which is no longer a thing).
  2. It runs the n-api-meta service rather than using config drive.
  3. It uses postgresql for the database.

So #1 is gone, and for #3, according to the April 2016 user survey (page
40) [1], 4% of reporting deployments are using it in production.

I don't think we're running n-api-meta in any other integrated gate
jobs, but I'm pretty sure there is at least one neutron job out there
that's running with it that way. We could also consider making the
nova-net dsvm full gate job run n-api-meta, or vice-versa with the
neutron dsvm full gate job.

We also have to consider that with HP public cloud being gone as a node
provider and we've got fewer test nodes to run with, we have to make
tough decisions about which jobs we're going to run in the integrated gate.

I'm bringing this up again because Nova has a few more jobs it would
like to make voting on it's repo (neutron LB and live migration, at
least in the check queue) but there are concerns about adding yet more
jobs that each change has to get through before it's merged, which means
if anything goes wrong in any of those we can have a 24 hour turnaround
on getting an approved change back through the gate.

[1]
https://www.openstack.org/assets/survey/April-2016-User-Survey-Report.pdf

Perhaps a better option would be to get oslo.db to run cross-project
checks like we do in requirements. That way the right team is covering
the usage of postgres and we still have coverage while still lowering
gate load for most projects.

I would support functional test jobs, but if a project that uses
oslo.db isn't going to also have the postgresql tests in place then
something checked in there could possibly break the ability to land
patches in oslo.db, so I don't think a one-directional "cross" project
test job is a good idea.

Doug


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Aug 18, 2016 by Doug_Hellmann (87,520 points)   3 4 9
0 votes

On 8/18/2016 1:18 PM, Matthew Thode wrote:
Perhaps a better option would be to get oslo.db to run cross-project
checks like we do in requirements. That way the right team is covering
the usage of postgres and we still have coverage while still lowering
gate load for most projects.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

I don't see the value in this unless there are projects that have
pg-specific code in them. The reason we have cross-project unit test
jobs for reqs changes is requirements changes in upper-constraints can
break and wedge the gate for a project, or multiple project. E.g.
removing something in a backward incompatible way, or the project with
the unit test is mocking something out poorly (like we've seen lately
with nova and python-neutronclient releases).

--

Thanks,

Matt Riedemann


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Aug 18, 2016 by Matt_Riedemann (48,320 points)   3 7 21
0 votes

On 08/18/2016 01:50 PM, Matt Riedemann wrote:
On 8/18/2016 1:18 PM, Matthew Thode wrote:

Perhaps a better option would be to get oslo.db to run cross-project
checks like we do in requirements. That way the right team is covering
the usage of postgres and we still have coverage while still lowering
gate load for most projects.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

I don't see the value in this unless there are projects that have
pg-specific code in them. The reason we have cross-project unit test
jobs for reqs changes is requirements changes in upper-constraints can
break and wedge the gate for a project, or multiple project. E.g.
removing something in a backward incompatible way, or the project with
the unit test is mocking something out poorly (like we've seen lately
with nova and python-neutronclient releases).

That makes sense, just improving the oslo.db test coverage for postgres
(if that's even necessary) would be good. The only other thing I'd like
to see (and it may already be done) is to have pg upgrade test coverage,
aka, I don't want to hit that keystone bug again :P But that's a
different conversation.

--
-- Matthew Thode (prometheanfire)


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

responded Aug 18, 2016 by prometheanfire_at_ge (6,880 points)   1 4 5
0 votes

On 08/18/2016 03:31 PM, Matthew Thode wrote:
On 08/18/2016 01:50 PM, Matt Riedemann wrote:

On 8/18/2016 1:18 PM, Matthew Thode wrote:

Perhaps a better option would be to get oslo.db to run cross-project
checks like we do in requirements. That way the right team is covering
the usage of postgres and we still have coverage while still lowering
gate load for most projects.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

I don't see the value in this unless there are projects that have
pg-specific code in them. The reason we have cross-project unit test
jobs for reqs changes is requirements changes in upper-constraints can
break and wedge the gate for a project, or multiple project. E.g.
removing something in a backward incompatible way, or the project with
the unit test is mocking something out poorly (like we've seen lately
with nova and python-neutronclient releases).

That makes sense, just improving the oslo.db test coverage for postgres
(if that's even necessary) would be good. The only other thing I'd like
to see (and it may already be done) is to have pg upgrade test coverage,
aka, I don't want to hit that keystone bug again :P But that's a
different conversation.

That's entirely doable inside the project. We do that in nova in our
unit tests. The important thing there is to not just run the schema
upgrades, but do them with some representative data in the tables.

-Sean

--
Sean Dague
http://dague.net


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Aug 18, 2016 by Sean_Dague (66,200 points)   4 8 14
0 votes

On Fri, Aug 19, 2016 at 1:00 AM, Matt Riedemann mriedem@linux.vnet.ibm.com
wrote:

It's that time of year again to talk about killing this job, at least from
the integrated gate (move it to experimental for people that care about
postgresql, or make it gating on a smaller subset of projects like oslo.db).

The postgresql job used to have three interesting things about it:

  1. It ran keystone with eventlet (which is no longer a thing).
  2. It runs the n-api-meta service rather than using config drive.
  3. It uses postgresql for the database.

So #1 is gone, and for #3, according to the April 2016 user survey (page
40) [1], 4% of reporting deployments are using it in production.

I don't think we're running n-api-meta in any other integrated gate jobs,
but I'm pretty sure there is at least one neutron job out there that's
running with it that way. We could also consider making the nova-net dsvm
full gate job run n-api-meta, or vice-versa with the neutron dsvm full gate
job.

We do now have functional testing of the metadata server though, so perhaps
that counts as coverage here?

We also have to consider that with HP public cloud being gone as a node
provider and we've got fewer test nodes to run with, we have to make tough
decisions about which jobs we're going to run in the integrated gate.

I'm bringing this up again because Nova has a few more jobs it would like
to make voting on it's repo (neutron LB and live migration, at least in the
check queue) but there are concerns about adding yet more jobs that each
change has to get through before it's merged, which means if anything goes
wrong in any of those we can have a 24 hour turnaround on getting an
approved change back through the gate.

[1] https://www.openstack.org/assets/survey/April-2016-User-Surv
ey-Report.pdf

Michael

--
Rackspace Australia


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Aug 18, 2016 by Michael_Still (16,180 points)   3 6 13
...