settingsLogin | Registersettings

[openstack-dev] [tempest][SR-IOV] tempest breaks Mellanox CI

0 votes

Hi all,

A recent change [1] in tempest broke all Mellanox CIs.

This is the second time it happened.

After the first time it happened we decided that Mellanox CI will comment on tempest.

On this time I saw that Mellanox CI was commenting on that patch with a failure but was still got approved - [2] Enabling Mellanox CI as commenting on tempest requires us physical resources such as servers/NICS because it tests SR-IOV.

So I am wandering what can be done in the future to prevent this from happen again.

Anyway we proposed the following fix to tempest [3]

[1] - https://review.openstack.org/#/c/320495/

[2] - http://13.69.151.247/95/320495/20/check-sriov-tempest/Tempest-Sriov/825304c/testr_results.html.gz

[3] - https://review.openstack.org/#/c/330331/


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Jun 16, 2016 in openstack-dev by Moshe_Levi (2,920 points)   3 6
retagged Jan 25, 2017 by admin

2 Responses

0 votes

On Thu, Jun 16, 2016 at 11:16 AM Moshe Levi moshele@mellanox.com wrote:

Hi all,

A recent change [1] in tempest broke all Mellanox CIs.

This is the second time it happened.

After the first time it happened we decided that Mellanox CI will
comment on tempest.

On this time I saw that Mellanox CI was commenting on that patch with a
failure but was still got approved - [2] Enabling Mellanox CI as commenting
on tempest requires us physical resources such as servers/NICS because
it tests SR-IOV.

So I am wandering what can be done in the future to prevent this from
happen again.

A message to the DL is a good way to raise awareness on this.
Ensuring the job stability is also important, too many failures will
decrease the attention on it - that said I don't have numbers for Mellanox
CI failure rates, so it may be that the pass rate is already good enough.

Anyway we proposed the following fix to tempest [3]

Thanks for fixing this, and sorry about the inconvenience.
Even with the fix in, the Mellanox CI job failed, which if unfortunate.
I issued a re-check.

Andrea Frittoli


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jun 16, 2016 by Andrea_Frittoli (5,920 points)   1 2 3
0 votes

Thanks,
We are investigating the failure.

From: Andrea Frittoli [mailto:andrea.frittoli@gmail.com]
Sent: Thursday, June 16, 2016 2:02 PM
To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [tempest][SR-IOV] tempest breaks Mellanox CI

On Thu, Jun 16, 2016 at 11:16 AM Moshe Levi moshele@mellanox.com wrote:

Hi all,

A recent change [1] in tempest broke all Mellanox CIs.

This is the second time it happened.

After the first time it happened we decided that Mellanox CI will comment on tempest.

On this time I saw that Mellanox CI was commenting on that patch with a failure but was still got approved - [2] Enabling Mellanox CI as commenting on tempest requires us physical resources such as servers/NICS because it tests SR-IOV.

So I am wandering what can be done in the future to prevent this from happen again.
A message to the DL is a good way to raise awareness on this.
Ensuring the job stability is also important, too many failures will decrease the attention on it - that said I don't have numbers for Mellanox CI failure rates, so it may be that the pass rate is already good enough.

Anyway we proposed the following fix to tempest [3]
Thanks for fixing this, and sorry about the inconvenience.
Even with the fix in, the Mellanox CI job failed, which if unfortunate.
I issued a re-check.

Andrea Frittoli

[1] - https://review.openstack.org/#/c/320495/

[2] - http://13.69.151.247/95/320495/20/check-sriov-tempest/Tempest-Sriov/825304c/testr_results.html.gz

[3] - https://review.openstack.org/#/c/330331/


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jun 16, 2016 by Lenny_Verkhovsky (1,460 points)   3
...