settingsLogin | Registersettings

[openstack-dev] [neutron] CI jobs take pretty long, can we improve that?

0 votes

Hello all,

the tests that we run on the gate for Neutron take pretty long (longer
than one hour). I think we can improve that and make better use of the
resources.
Here are some ideas that came up when Ihar and I discussed this topic
during the sprint in Brno:

1) We have few jobs that are non-voting. I think it's OK to have
non-voting jobs for a limited amount of time, while we try to make them
stable but this shouldn't be too long, otherwise we waste time running
those tests without even using the results. If a job is still not-voting
after 3 months (or 4 or 6, we can find a good time interval) the job
should be removed. My hope is that this threat will make us find some
time to actually fix the job and make it vote :)

2) multi-node jobs run for every patch set. Is that really what we want?
They take pretty long. We could move them to a periodic job. I know we
can easily forget about periodic jobs, to avoid that we could run them
in the gate queue too. If a patch can't merge because of a failure we
will fix the issue. To trigger them for a specific patch that might
affect multi-node we can run the experimental jobs.

Thoughts?

Rossella


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Mar 21, 2016 in openstack-dev by Rossella_Sblendido (2,660 points)   1 3

18 Responses

0 votes

Rossella Sblendido wrote:
2) multi-node jobs run for every patch set. Is that really what we want?
They take pretty long. We could move them to a periodic job.

I would rather remove all the single-node jobs. Nova has been moving to
multinode jobs for their gate (if I recall correctly my
conversation with Dan Smith) and we should be moving in this direction
too. We should test Neutron the way it is deployed in production.

Also, who is really monitoring the periodic jobs? Truthfully? I know
there are some IPv6 jobs that are periodic and I'll be the first to
admit that I am not following them at all.

So, my thinking is, unless it's running at the gate and inflicting pain
on people, it's not going to be a treated as a priority. Look at Linux
Bridge - serious race conditions that existed for years only
got fixed once I inflicted pain on all the Neutron devs by making it
voting and running on every patchset (sorry, not sorry).

--
Sean M. Collins


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 21, 2016 by Sean_M._Collins (11,480 points)   5 8 14
0 votes

Sean M. Collins sean@coreitpro.com wrote:

Rossella Sblendido wrote:

2) multi-node jobs run for every patch set. Is that really what we want?
They take pretty long. We could move them to a periodic job.

I would rather remove all the single-node jobs. Nova has been moving to
multinode jobs for their gate (if I recall correctly my
conversation with Dan Smith) and we should be moving in this direction
too. We should test Neutron the way it is deployed in production.

Also, who is really monitoring the periodic jobs? Truthfully? I know
there are some IPv6 jobs that are periodic and I'll be the first to
admit that I am not following them at all.

Well, stable maintainers track their periodic job failures. :) Email
notifications when something starts failing help.

So, my thinking is, unless it's running at the gate and inflicting pain
on people, it's not going to be a treated as a priority. Look at Linux
Bridge - serious race conditions that existed for years only
got fixed once I inflicted pain on all the Neutron devs by making it
voting and running on every patchset (sorry, not sorry).

I think there is still common ground between you and Rossella’s stances:
the fact that we want to inflict gating pain does not mean that we want to
execute every single job on each PS uploaded to gerrit. For some advanced
and non-obvious checks [like partial grenade] the validation could be
probably postponed till the patch hits the gate.

Yes, sometimes it will mean gate being reset due to the bad patch. This can
be avoided in most of cases if reviewers and the author for a patch that
potentially touches a specific scenario execute the jobs before hitting the
gate with the patch [for example, if the job is in experimental set, it’s a
matter of ‘check experimental’ before pressing W+1].

Ihar


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 21, 2016 by Ihar_Hrachyshka (35,300 points)   3 11 19
0 votes

On Mar 21, 2016, at 5:40 AM, Ihar Hrachyshka ihrachys@redhat.com wrote:

Sean M. Collins sean@coreitpro.com wrote:

Rossella Sblendido wrote:

2) multi-node jobs run for every patch set. Is that really what we want?
They take pretty long. We could move them to a periodic job.

I would rather remove all the single-node jobs. Nova has been moving to
multinode jobs for their gate (if I recall correctly my
conversation with Dan Smith) and we should be moving in this direction
too. We should test Neutron the way it is deployed in production.

Also, who is really monitoring the periodic jobs? Truthfully? I know
there are some IPv6 jobs that are periodic and I'll be the first to
admit that I am not following them at all.

Well, stable maintainers track their periodic job failures. :) Email notifications when something starts failing help.

So, my thinking is, unless it's running at the gate and inflicting pain
on people, it's not going to be a treated as a priority. Look at Linux
Bridge - serious race conditions that existed for years only
got fixed once I inflicted pain on all the Neutron devs by making it
voting and running on every patchset (sorry, not sorry).

I think there is still common ground between you and Rossella’s stances: the fact that we want to inflict gating pain does not mean that we want to execute every single job on each PS uploaded to gerrit. For some advanced and non-obvious checks [like partial grenade] the validation could be probably postponed till the patch hits the gate.

Yes, sometimes it will mean gate being reset due to the bad patch. This can be avoided in most of cases if reviewers and the author for a patch that potentially touches a specific scenario execute the jobs before hitting the gate with the patch [for example, if the job is in experimental set, it’s a matter of ‘check experimental’ before pressing W+1].

We have been pretty consciously moving neutron jobs to cause pain to neutron and not everyone else, which is the opposite of a “gate only” plan. Aside from that being against infra policy, I think I’m reading between the lines that folks are wanting faster iterations between patchsets. I note that the standard -full job is up to 55-65 minutes, from it’s old time of 40-45. Have we characterized why that’s so much slower now? Perhaps addressing that will bring down to the turn-around for all.

Thanks,
doug

Ihar


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 21, 2016 by dougwig_at_parksides (7,230 points)   1 2 5
0 votes

On 21 March 2016 at 04:32, Sean M. Collins sean@coreitpro.com wrote:

Rossella Sblendido wrote:

2) multi-node jobs run for every patch set. Is that really what we want?
They take pretty long. We could move them to a periodic job.

I would rather remove all the single-node jobs. Nova has been moving to
multinode jobs for their gate (if I recall correctly my
conversation with Dan Smith) and we should be moving in this direction
too. We should test Neutron the way it is deployed in production.

This was not true last time I checked. Switching to multinode jobs for the
gate means that all projects in the integrated gate will have to use the
miltinode configuration.

Also, who is really monitoring the periodic jobs? Truthfully? I know
there are some IPv6 jobs that are periodic and I'll be the first to
admit that I am not following them at all.

So, my thinking is, unless it's running at the gate and inflicting pain
on people, it's not going to be a treated as a priority. Look at Linux
Bridge - serious race conditions that existed for years only
got fixed once I inflicted pain on all the Neutron devs by making it
voting and running on every patchset (sorry, not sorry).

--
Sean M. Collins


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 21, 2016 by Armando_M. (23,560 points)   2 6 9
0 votes

On 21 March 2016 at 04:15, Rossella Sblendido rsblendido@suse.com wrote:

Hello all,

the tests that we run on the gate for Neutron take pretty long (longer
than one hour). I think we can improve that and make better use of the
resources.

Here are some ideas that came up when Ihar and I discussed this topic
during the sprint in Brno:

1) We have few jobs that are non-voting. I think it's OK to have
non-voting jobs for a limited amount of time, while we try to make them
stable but this shouldn't be too long, otherwise we waste time running
those tests without even using the results. If a job is still not-voting
after 3 months (or 4 or 6, we can find a good time interval) the job
should be removed. My hope is that this threat will make us find some
time to actually fix the job and make it vote :)

2) multi-node jobs run for every patch set. Is that really what we want?
They take pretty long. We could move them to a periodic job. I know we
can easily forget about periodic jobs, to avoid that we could run them
in the gate queue too. If a patch can't merge because of a failure we
will fix the issue. To trigger them for a specific patch that might
affect multi-node we can run the experimental jobs.

Thoughts?

Thanks for raising the topic. That said, I am not sure I see how what you
propose is going to make things better. Jobs, either non voting or multnode
run in parallel, thus reducing the number of jobs won't reduce the time to
feedback though it would improve resource usage. We are already pretty
conscious of that and compared to other projects we already run a limited
numbers of jobs, but we can do better, of course.

Do you have an a better insight of job runtimes vs jobs in other projects?
Most of the time in the job runtime is actually spent setting the
infrastructure up, and I am not sure we can do anything about it, unless we
take this with Infra.

Rossella


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 21, 2016 by Armando_M. (23,560 points)   2 6 9
0 votes

On Mon, Mar 21, 2016, at 09:32 AM, Armando M. wrote:
Do you have an a better insight of job runtimes vs jobs in other
projects?
Most of the time in the job runtime is actually spent setting the
infrastructure up, and I am not sure we can do anything about it, unless
we
take this with Infra.

I haven't done a comparison yet buts lets break down the runtime of a
recent successful neutron full run against neutron master [0].

Basic host setup takes 65 seconds. Start of job to 2016-03-17
22:14:27.397 [1]
Workspace setup takes 520 seconds. 2016-03-17 22:14:27.397 [1] to
2016-03-17 22:23:07.429 [2]
Devstack takes 1205 seconds. 2016-03-17 22:23:18.760 [3] to 2016-03-17
22:43:23.339 [4]
Loading old tempest subunit streams takes 155 seconds. 2016-03-17
22:43:23.340 [5] to 2016-03-17 22:45:58.061 [6]
Tempest takes 1982 seconds. 2016-03-17 22:45:58.201 [7] to 2016-03-17
23:19:00.117 [8]
Then we spend the rest of the test time (76 seconds) cleaning up.
2016-03-17 23:19:00.117 [8] to end of job.

Note that I haven't accounted for all of the time used and instead of
focused on the major steps that use the most time. Also it is Monday
morning and some of my math may be off.

[0]
http://logs.openstack.org/18/294018/3/check/gate-tempest-dsvm-neutron-full/1cce1e8/
[1]
http://logs.openstack.org/18/294018/3/check/gate-tempest-dsvm-neutron-full/1cce1e8/console.html.gz#_2016-03-17_22_14_27_397
[2]
http://logs.openstack.org/18/294018/3/check/gate-tempest-dsvm-neutron-full/1cce1e8/console.html.gz#_2016-03-17_22_23_07_429
[3]
http://logs.openstack.org/18/294018/3/check/gate-tempest-dsvm-neutron-full/1cce1e8/console.html.gz#_2016-03-17_22_23_18_760
[4]
http://logs.openstack.org/18/294018/3/check/gate-tempest-dsvm-neutron-full/1cce1e8/console.html.gz#_2016-03-17_22_43_23_339
[5]
http://logs.openstack.org/18/294018/3/check/gate-tempest-dsvm-neutron-full/1cce1e8/console.html.gz#_2016-03-17_22_43_23_340
[6]
http://logs.openstack.org/18/294018/3/check/gate-tempest-dsvm-neutron-full/1cce1e8/console.html.gz#_2016-03-17_22_45_58_061
[7]
http://logs.openstack.org/18/294018/3/check/gate-tempest-dsvm-neutron-full/1cce1e8/console.html.gz#_2016-03-17_22_45_58_201
[8]
http://logs.openstack.org/18/294018/3/check/gate-tempest-dsvm-neutron-full/1cce1e8/console.html.gz#_2016-03-17_23_19_00_117

One big takeaway from this is that the vast majority of the time is
spent in devstack and tempest not in the infrastructure setup. You
should be able to dig into both the devstack setup and tempest test
runtimes and hopefully speed things up.

Hopefully this gives you enough information to get started into digging
on this.

Clark


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 21, 2016 by Clark_Boylan (8,800 points)   1 2 4
0 votes

On 21 March 2016 at 11:08, Clark Boylan cboylan@sapwetik.org wrote:

On Mon, Mar 21, 2016, at 09:32 AM, Armando M. wrote:

Do you have an a better insight of job runtimes vs jobs in other
projects?
Most of the time in the job runtime is actually spent setting the
infrastructure up, and I am not sure we can do anything about it, unless
we
take this with Infra.

I haven't done a comparison yet buts lets break down the runtime of a
recent successful neutron full run against neutron master [0].

Basic host setup takes 65 seconds. Start of job to 2016-03-17
22:14:27.397 [1]
Workspace setup takes 520 seconds. 2016-03-17 22:14:27.397 [1] to
2016-03-17 22:23:07.429 [2]
Devstack takes 1205 seconds. 2016-03-17 22:23:18.760 [3] to 2016-03-17
22:43:23.339 [4]
Loading old tempest subunit streams takes 155 seconds. 2016-03-17
22:43:23.340 [5] to 2016-03-17 22:45:58.061 [6]
Tempest takes 1982 seconds. 2016-03-17 22:45:58.201 [7] to 2016-03-17
23:19:00.117 [8]
Then we spend the rest of the test time (76 seconds) cleaning up.
2016-03-17 23:19:00.117 [8] to end of job.

Note that I haven't accounted for all of the time used and instead of
focused on the major steps that use the most time. Also it is Monday
morning and some of my math may be off.

[0]

http://logs.openstack.org/18/294018/3/check/gate-tempest-dsvm-neutron-full/1cce1e8/
[1]

http://logs.openstack.org/18/294018/3/check/gate-tempest-dsvm-neutron-full/1cce1e8/console.html.gz#_2016-03-17_22_14_27_397
[2]

http://logs.openstack.org/18/294018/3/check/gate-tempest-dsvm-neutron-full/1cce1e8/console.html.gz#_2016-03-17_22_23_07_429
[3]

http://logs.openstack.org/18/294018/3/check/gate-tempest-dsvm-neutron-full/1cce1e8/console.html.gz#_2016-03-17_22_23_18_760
[4]

http://logs.openstack.org/18/294018/3/check/gate-tempest-dsvm-neutron-full/1cce1e8/console.html.gz#_2016-03-17_22_43_23_339
[5]

http://logs.openstack.org/18/294018/3/check/gate-tempest-dsvm-neutron-full/1cce1e8/console.html.gz#_2016-03-17_22_43_23_340
[6]

http://logs.openstack.org/18/294018/3/check/gate-tempest-dsvm-neutron-full/1cce1e8/console.html.gz#_2016-03-17_22_45_58_061
[7]

http://logs.openstack.org/18/294018/3/check/gate-tempest-dsvm-neutron-full/1cce1e8/console.html.gz#_2016-03-17_22_45_58_201
[8]

http://logs.openstack.org/18/294018/3/check/gate-tempest-dsvm-neutron-full/1cce1e8/console.html.gz#_2016-03-17_23_19_00_117

One big takeaway from this is that the vast majority of the time is
spent in devstack and tempest not in the infrastructure setup. You
should be able to dig into both the devstack setup and tempest test
runtimes and hopefully speed things up.

Hopefully this gives you enough information to get started into digging
on this.

Clark: thanks for this insightful response.

I should clarify my comment about infrastructure setup (it is Monday for me
too :)): what I meant was the there is a good portion of time spent to get
to a point where tests can be run. That includes node setup as well as
stacking. That is obviously less than 50%, but even >30% feels like a
substantial overhead. I am not sure what we can do about it, but looping
you in this discussion seemed like the least this thread should do.

That said, there are many tempest tests that take over 30 seconds to
complete and those do not even touch Neutron. For those that do, then we
should clearly identify where the slowness comes from and I think that's
where, as a Neutron team, our focus should be.

IMO, before we go on and talk about evicting jobs, I think we should take a
closer look (i.e. profiling) where time is spent so that we can make each
test run leaner.

[1]
http://status.openstack.org//openstack-health/#/job/gate-tempest-dsvm-neutron-full?groupKey=project&resolutionKey=hour&end=2016-03-21T18:14:19.534Z

Clark


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 21, 2016 by Armando_M. (23,560 points)   2 6 9
0 votes

On Mon, Mar 21, 2016, at 11:08 AM, Clark Boylan wrote:
On Mon, Mar 21, 2016, at 09:32 AM, Armando M. wrote:

Do you have an a better insight of job runtimes vs jobs in other
projects?
Most of the time in the job runtime is actually spent setting the
infrastructure up, and I am not sure we can do anything about it, unless
we
take this with Infra.

I haven't done a comparison yet buts lets break down the runtime of a
recent successful neutron full run against neutron master [0].

And now for some comparative data from the gate-tempest-dsvm-full job
[0]. This job also ran against a master change that merged and ran in
the same cloud and region as the neutron job.

Basic host setup takes 63 seconds. Start of job to 2016-03-21
16:46:41.058 [1]
Workspace setup takes 380 seconds. 2016-03-21 16:46:41.058 [1] to
2016-03-21 16:53:01.754 [2]
Devstack takes 890 seconds. 2016-03-21 16:53:19.235 [3] to 2016-03-21
17:08:10.082 [4]
Loading old tempest subunit streams takes 63 seconds. 2016-03-21
17:08:10.111 [5] to 2016-03-21 17:09:13.454 [6]
Tempest takes 1347 seconds. 2016-03-21 17:09:13.587 [7] to 2016-03-21
17:31:40.885 [8]
Then we spend the rest of the test time (52 seconds) cleaning up.
2016-03-21 17:31:40.885 [8] to end of job.

[0]
http://logs.openstack.org/48/294548/1/gate/gate-tempest-dsvm-full/d94901e/
[1]
http://logs.openstack.org/48/294548/1/gate/gate-tempest-dsvm-full/d94901e/console.html#_2016-03-21_16_46_41_058
[2]
http://logs.openstack.org/48/294548/1/gate/gate-tempest-dsvm-full/d94901e/console.html#_2016-03-21_16_53_01_754
[3]
http://logs.openstack.org/48/294548/1/gate/gate-tempest-dsvm-full/d94901e/console.html#_2016-03-21_16_53_19_235
[4]
http://logs.openstack.org/48/294548/1/gate/gate-tempest-dsvm-full/d94901e/console.html#_2016-03-21_17_08_10_082
[5]
http://logs.openstack.org/48/294548/1/gate/gate-tempest-dsvm-full/d94901e/console.html#_2016-03-21_17_08_10_111
[6]
http://logs.openstack.org/48/294548/1/gate/gate-tempest-dsvm-full/d94901e/console.html#_2016-03-21_17_09_13_454
[7]
http://logs.openstack.org/48/294548/1/gate/gate-tempest-dsvm-full/d94901e/console.html#_2016-03-21_17_09_13_587
[8]
http://logs.openstack.org/48/294548/1/gate/gate-tempest-dsvm-full/d94901e/console.html#_2016-03-21_17_31_40_885

Generally each step of this job was quicker. There were big differences
in devstack and tempest run time though. Is devstack much slower to
setup neutron when compared to nova net? For tempest it looks like we
run ~1510 tests against neutron and only ~1269 against nova net. This
may account for the large difference there. I also recall that we run
ipv6 tempest tests against neutron deployments that were inefficient and
booted 2 qemu VMs per test (not sure if that is still the case but
illustrates that the tests themselves may not be very quick in the
neutron case).

Of course we may also be seeing differences in cloud VMs (though tried
to control for by looking at tests that ran in the same region). Hard to
say without more data. In any case this hopefully serves as a good
starting point for others to dig into the ~20 minute discrepancy between
nova net + tempest and neutron + tempest.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 21, 2016 by Clark_Boylan (8,800 points)   1 2 4
0 votes

On Mon, Mar 21, 2016, at 11:49 AM, Clark Boylan wrote:
On Mon, Mar 21, 2016, at 11:08 AM, Clark Boylan wrote:

On Mon, Mar 21, 2016, at 09:32 AM, Armando M. wrote:

Do you have an a better insight of job runtimes vs jobs in other
projects?
Most of the time in the job runtime is actually spent setting the
infrastructure up, and I am not sure we can do anything about it, unless
we
take this with Infra.

I haven't done a comparison yet buts lets break down the runtime of a
recent successful neutron full run against neutron master [0].

And now for some comparative data from the gate-tempest-dsvm-full job
[0]. This job also ran against a master change that merged and ran in
the same cloud and region as the neutron job.

snip

Generally each step of this job was quicker. There were big differences
in devstack and tempest run time though. Is devstack much slower to
setup neutron when compared to nova net? For tempest it looks like we
run ~1510 tests against neutron and only ~1269 against nova net. This
may account for the large difference there. I also recall that we run
ipv6 tempest tests against neutron deployments that were inefficient and
booted 2 qemu VMs per test (not sure if that is still the case but
illustrates that the tests themselves may not be very quick in the
neutron case).

Looking at the tempest slowest tests output for each of these jobs
(neutron and nova net) some tests line up really well across jobs and
others do not. In order to get a better handle on the runtime for
individual tests I have pushed https://review.openstack.org/295487 which
will run tempest serially reducing the competition for resources between
tests.

Hopefully the subunit logs generated by this change can provide more
insight into where we are losing time during the tempest test runs.

Clark


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 21, 2016 by Clark_Boylan (8,800 points)   1 2 4
0 votes

On 03/21/2016 04:09 PM, Clark Boylan wrote:
On Mon, Mar 21, 2016, at 11:49 AM, Clark Boylan wrote:

On Mon, Mar 21, 2016, at 11:08 AM, Clark Boylan wrote:

On Mon, Mar 21, 2016, at 09:32 AM, Armando M. wrote:

Do you have an a better insight of job runtimes vs jobs in other
projects?
Most of the time in the job runtime is actually spent setting the
infrastructure up, and I am not sure we can do anything about it, unless
we
take this with Infra.

I haven't done a comparison yet buts lets break down the runtime of a
recent successful neutron full run against neutron master [0].

And now for some comparative data from the gate-tempest-dsvm-full job
[0]. This job also ran against a master change that merged and ran in
the same cloud and region as the neutron job.

snip

Generally each step of this job was quicker. There were big differences
in devstack and tempest run time though. Is devstack much slower to
setup neutron when compared to nova net? For tempest it looks like we
run ~1510 tests against neutron and only ~1269 against nova net. This
may account for the large difference there. I also recall that we run
ipv6 tempest tests against neutron deployments that were inefficient and
booted 2 qemu VMs per test (not sure if that is still the case but
illustrates that the tests themselves may not be very quick in the
neutron case).

Looking at the tempest slowest tests output for each of these jobs
(neutron and nova net) some tests line up really well across jobs and
others do not. In order to get a better handle on the runtime for
individual tests I have pushed https://review.openstack.org/295487 which
will run tempest serially reducing the competition for resources between
tests.

Hopefully the subunit logs generated by this change can provide more
insight into where we are losing time during the tempest test runs.

Subunit logs aren't the full story here. Activity in addCleanup doesn't
get added to the subunit time accounting for the test, which causes some
interesting issues when waiting for resources to delete. I would be
especially cautious of that on some of these.

-Sean

--
Sean Dague
http://dague.net


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 21, 2016 by Sean_Dague (66,200 points)   4 12 22
...