settingsLogin | Registersettings

[openstack-dev] [infra] [cinder] CI via infra for the DRBD Cinder driver

0 votes

Hi all,

this is a reflection of the discussion I just had on #openstack-infra; it's
about (re-)using the central CI infrastructure for our Open-Source DRBD
driver too.

The current status is:
* The DRBD driver is already in Cinder, so DRBD-replicated Cinder storage
using iSCSI to the hypervisors does work out-of-the-box.
* The Nova-parts didn't make it in for Kilo; we'll try to get them into L.
* I've got a lib/backends/drbd for devstack, that together with a matching
local.conf can set up a node - at least for a limited set of
distributions (as DRBD needs a kernel module, Ubuntu/Debian via DKMS are
the easy way).
[Please note that package installation is not yet done in this script
yet - I'm not sure whether I can/may/should simply add an
apt-repository.]

Now, clarkb told me about two caveats:

«Yup, so the two things I will start with is that multinode testing is
still really rudimentary, we only just got tempest sort of working with
it. So I might suggest running on a single node first to get the general
thing working.

The other thing is that we don't have the zuul code to vote with
a different account deployed/merged yet. So initially you could run your
job but it wouldn't vote against, say, cinder.»

Cinder has a deadline for CI: March 19th; upon relaying that fact (resp.
nearly correct date) clarkb said

«thats about 3 weeks... probably at least for the zuul thing.»

So, actually it's nearly 4 weeks, let's hope that it all works out.

Actually, the multi-node testing will only be needed when we get the Nova
parts in, because then it would make sense to test (Nova) via both iSCSI
and the DRBD transport; for Cinder CI a single-node setup is sufficient.

My remaining questions are:
* Is it possible to have our driver tested via the common infrastructure?
* Is it okay to setup another apt-repository during the devstack run,
to install the needed packages? I'm not sure whether our servers
would simply be accessible, some firewall or filtering proxy could
break such things easily.
* Apart from the cinder-backend script in devstack (which I'll have to
finish first, see eg. package installation), is any other information
needed from us?

Thank you for your feedback and any help you can offer!

Regards,

Phil

--
: Ing. Philipp Marek
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com :

DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Feb 20, 2015 in openstack-dev by Philipp_Marek (1,760 points)   1 4
retagged Mar 6, 2015 by admin

9 Responses

0 votes

On Fri, Feb 20, 2015, at 11:24 AM, Philipp Marek wrote:
Hi all,

this is a reflection of the discussion I just had on #openstack-infra;
it's
about (re-)using the central CI infrastructure for our Open-Source DRBD
driver too.

The current status is:
* The DRBD driver is already in Cinder, so DRBD-replicated Cinder
storage
using iSCSI to the hypervisors does work out-of-the-box.
* The Nova-parts didn't make it in for Kilo; we'll try to get them into
L.
* I've got a lib/backends/drbd for devstack, that together with a
matching
local.conf can set up a node - at least for a limited set of
distributions (as DRBD needs a kernel module, Ubuntu/Debian via DKMS
are
the easy way).
[Please note that package installation is not yet done in this script
yet - I'm not sure whether I can/may/should simply add an
apt-repository.]

Now, clarkb told me about two caveats:

«Yup, so the two things I will start with is that multinode testing is
still really rudimentary, we only just got tempest sort of working
with
it. So I might suggest running on a single node first to get the
general
thing working.
You can follow the current state of multinode work at
https://review.openstack.org/#/c/141530/

The other thing is that we don't have the zuul code to vote with
a different account deployed/merged yet. So initially you could run
your
job but it wouldn't vote against, say, cinder.»
The stack that adds the necessary zuul stuff ends with
https://review.openstack.org/#/c/121528/

Cinder has a deadline for CI: March 19th; upon relaying that fact (resp.
nearly correct date) clarkb said

«thats about 3 weeks... probably at least for the zuul thing.»

So, actually it's nearly 4 weeks, let's hope that it all works out.

Actually, the multi-node testing will only be needed when we get the Nova
parts in, because then it would make sense to test (Nova) via both iSCSI
and the DRBD transport; for Cinder CI a single-node setup is sufficient.

My remaining questions are:
* Is it possible to have our driver tested via the common
infrastructure?
I think it should be, initially just reporting against your devstack
plugin while things get sorted out then once zuul has the appropriate
bits reporting like a third party test account but running upstream.
* Is it okay to setup another apt-repository during the devstack run,
to install the needed packages? I'm not sure whether our servers
would simply be accessible, some firewall or filtering proxy could
break such things easily.
It is not ok in a gating job to do that but that isn't your intention so
should be fine. That said, isn't drbd available in existing apt/yum
repos? What special sauce do you need to pull in?
* Apart from the cinder-backend script in devstack (which I'll have to
finish first, see eg. package installation), is any other information
needed from us?
Other than following this thread and writing that devstack plugin
probably not.

Hope this helps,
Clark


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Feb 23, 2015 by Clark_Boylan (8,800 points)   1 2 4
0 votes

Clark Boylan cboylan@sapwetik.org writes:

The other thing is that we don't have the zuul code to vote with
a different account deployed/merged yet. So initially you could run
your
job but it wouldn't vote against, say, cinder.»
The stack that adds the necessary zuul stuff ends with
https://review.openstack.org/#/c/121528/

I don't believe that we have any current plans to use that code in
infra. I don't believe that it makes sense for us to create multiple
accounts in Gerrit for our single system. It's quite a bit of overhead,
and I don't believe it is necessary.

To be clear, I think any policy that says that drivers must have
"third-party CI" is an oversight. I believe that it's fine to require
them to have "CI", but if the CI can be provided by infra, it should be.
I believe this misunderstanding comes from the fact that most
out-of-tree drivers require specialized hardware, or non-free software,
that can not be run in our system.

As mentioned elsewhere, we're generally quite happy to have any open
source component tested in the upstream infrastructure. Since this
qualifies, I think the quickest and simplest way to proceed is to create
a job that runs on the driver repo, and then create a non-voting version
to run on the cinder repo.

Additionally, if it proves stable, the Cinder developers could certainly
choose to gate on this job as well. That's entirely up to them, but
there's no policy or technical reason it can not.

-Jim


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Feb 24, 2015 by James_E._Blair (5,080 points)   1 6 8
0 votes
  • Is it okay to setup another apt-repository during the devstack run,
    to install the needed packages? I'm not sure whether our servers
    would simply be accessible, some firewall or filtering proxy could
    break such things easily.
    It is not ok in a gating job to do that but that isn't your intention so
    should be fine. That said, isn't drbd available in existing apt/yum
    repos? What special sauce do you need to pull in?
    Well, neither DRBDmanage nor DRBD 9 (which have an 0.21 resp. an rc1 out
    now) are stable yet, and so are not yet available in any upstream
    repository.

We can either provide an apt repository or put the needed .debs into some
location, to have them installed during the devstack run.

I guess the later option is better because it won't have issues with
firewalls, proxies, etc., and we will only provide versions that proved
more-or-less stable anyway - there's no need to change these that often,
I guess that once every two weeks or so is a good approximation.

--
: Ing. Philipp Marek
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com :

DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Feb 24, 2015 by Philipp_Marek (1,760 points)   1 4
0 votes

On 02/23/2015 07:00 PM, James E. Blair wrote:
Clark Boylan cboylan@sapwetik.org writes:

The other thing is that we don't have the zuul code to vote with
a different account deployed/merged yet. So initially you could run
your
job but it wouldn't vote against, say, cinder.»
The stack that adds the necessary zuul stuff ends with
https://review.openstack.org/#/c/121528/

I don't believe that we have any current plans to use that code in
infra. I don't believe that it makes sense for us to create multiple
accounts in Gerrit for our single system. It's quite a bit of overhead,
and I don't believe it is necessary.

To be clear, I think any policy that says that drivers must have
"third-party CI" is an oversight. I believe that it's fine to require
them to have "CI", but if the CI can be provided by infra, it should be.
I believe this misunderstanding comes from the fact that most
out-of-tree drivers require specialized hardware, or non-free software,
that can not be run in our system.

As mentioned elsewhere, we're generally quite happy to have any open
source component tested in the upstream infrastructure. Since this
qualifies, I think the quickest and simplest way to proceed is to create
a job that runs on the driver repo, and then create a non-voting version
to run on the cinder repo.

Additionally, if it proves stable, the Cinder developers could certainly
choose to gate on this job as well.
I'd like to make sure that if Infra is saying that CI jobs on drivers
can gate that this is a substantial difference from what I have been
saying for a year, specifically that they can't and won't.

Every CI system/job author/operator that I have interacted with for the
past year has been trying to figure out how to be in the gate. Since
that implies a relationship in which development on the service can't
proceed without a driver's say so (by passing the test in the gate) this
is exactly the relationship I have been trying to ensure doesn't happen.
By stating that driver jobs can gate on the service, this opens the
flood gates of every CI operator increasing pressure to be in the gate
as well (the pressure never stopped, it just took different forms).

I disagree with the relationship where a service's development can only
proceed at the speed of a driver (or driver's because where there is
one, there are plenty more).

Thanks,
Anita.

That's entirely up to them, but
there's no policy or technical reason it can not.

-Jim


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Feb 24, 2015 by Anita_Kuno (21,320 points)   3 3 4
0 votes

On 24 February 2015 at 02:00, James E. Blair corvus@inaugust.com wrote:

Clark Boylan cboylan@sapwetik.org writes:

To be clear, I think any policy that says that drivers must have
"third-party CI" is an oversight. I believe that it's fine to require
them to have "CI", but if the CI can be provided by infra, it should be.
I believe this misunderstanding comes from the fact that most
out-of-tree drivers require specialized hardware, or non-free software,
that can not be run in our system.

From a cinder PoV, we don't care if infra is running the CI or somebody
else, it's all good. I think things are slightly less clear when jenkins is
running a mix of gating and none gating jobs, and it is slightly harder for
cinder core to know where to go to get a faulty CI job turned off, but I
believe infra can manage the latter concern.

So yeah, if infra are happy hosting the DRDB ci, then great.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Feb 24, 2015 by Duncan_Thomas (16,160 points)   1 3 6
0 votes

For cinder, we don't want to gate on drivers, and will be unhappy if that
gets turned on, so the story hasn't changed, Anita.

I think Jim was pointing out the technical possibility, which is a fine and
valid thing to point out. Cinder core are likely to continue not to want
it, for the foreseeable future.

On 24 February 2015 at 10:30, Anita Kuno anteaya@anteaya.info wrote:

On 02/23/2015 07:00 PM, James E. Blair wrote:

Clark Boylan cboylan@sapwetik.org writes:

The other thing is that we don't have the zuul code to vote with
a different account deployed/merged yet. So initially you could run
your
job but it wouldn't vote against, say, cinder.»
The stack that adds the necessary zuul stuff ends with
https://review.openstack.org/#/c/121528/

I don't believe that we have any current plans to use that code in
infra. I don't believe that it makes sense for us to create multiple
accounts in Gerrit for our single system. It's quite a bit of overhead,
and I don't believe it is necessary.

To be clear, I think any policy that says that drivers must have
"third-party CI" is an oversight. I believe that it's fine to require
them to have "CI", but if the CI can be provided by infra, it should be.
I believe this misunderstanding comes from the fact that most
out-of-tree drivers require specialized hardware, or non-free software,
that can not be run in our system.

As mentioned elsewhere, we're generally quite happy to have any open
source component tested in the upstream infrastructure. Since this
qualifies, I think the quickest and simplest way to proceed is to create
a job that runs on the driver repo, and then create a non-voting version
to run on the cinder repo.

Additionally, if it proves stable, the Cinder developers could certainly
choose to gate on this job as well.
I'd like to make sure that if Infra is saying that CI jobs on drivers
can gate that this is a substantial difference from what I have been
saying for a year, specifically that they can't and won't.

Every CI system/job author/operator that I have interacted with for the
past year has been trying to figure out how to be in the gate. Since
that implies a relationship in which development on the service can't
proceed without a driver's say so (by passing the test in the gate) this
is exactly the relationship I have been trying to ensure doesn't happen.
By stating that driver jobs can gate on the service, this opens the
flood gates of every CI operator increasing pressure to be in the gate
as well (the pressure never stopped, it just took different forms).

I disagree with the relationship where a service's development can only
proceed at the speed of a driver (or driver's because where there is
one, there are plenty more).

Thanks,
Anita.

That's entirely up to them, but
there's no policy or technical reason it can not.

-Jim


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Duncan Thomas


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Feb 24, 2015 by Duncan_Thomas (16,160 points)   1 3 6
0 votes

On 02/24/2015 04:40 AM, Duncan Thomas wrote:
For cinder, we don't want to gate on drivers, and will be unhappy if that
gets turned on, so the story hasn't changed, Anita.

I think Jim was pointing out the technical possibility, which is a fine and
valid thing to point out. Cinder core are likely to continue not to want
it, for the foreseeable future.
Fair enough.

My concern is that by making it technically possible and permissible
from infra's side we are just opening up the projects to have to engage
in yet another game of whack-a-mole since driver testing ops can and
will try to turn it on.

Thanks,
Anita.

On 24 February 2015 at 10:30, Anita Kuno anteaya@anteaya.info wrote:

On 02/23/2015 07:00 PM, James E. Blair wrote:

Clark Boylan cboylan@sapwetik.org writes:

The other thing is that we don't have the zuul code to vote with
a different account deployed/merged yet. So initially you could run
your
job but it wouldn't vote against, say, cinder.»
The stack that adds the necessary zuul stuff ends with
https://review.openstack.org/#/c/121528/

I don't believe that we have any current plans to use that code in
infra. I don't believe that it makes sense for us to create multiple
accounts in Gerrit for our single system. It's quite a bit of overhead,
and I don't believe it is necessary.

To be clear, I think any policy that says that drivers must have
"third-party CI" is an oversight. I believe that it's fine to require
them to have "CI", but if the CI can be provided by infra, it should be.
I believe this misunderstanding comes from the fact that most
out-of-tree drivers require specialized hardware, or non-free software,
that can not be run in our system.

As mentioned elsewhere, we're generally quite happy to have any open
source component tested in the upstream infrastructure. Since this
qualifies, I think the quickest and simplest way to proceed is to create
a job that runs on the driver repo, and then create a non-voting version
to run on the cinder repo.

Additionally, if it proves stable, the Cinder developers could certainly
choose to gate on this job as well.
I'd like to make sure that if Infra is saying that CI jobs on drivers
can gate that this is a substantial difference from what I have been
saying for a year, specifically that they can't and won't.

Every CI system/job author/operator that I have interacted with for the
past year has been trying to figure out how to be in the gate. Since
that implies a relationship in which development on the service can't
proceed without a driver's say so (by passing the test in the gate) this
is exactly the relationship I have been trying to ensure doesn't happen.
By stating that driver jobs can gate on the service, this opens the
flood gates of every CI operator increasing pressure to be in the gate
as well (the pressure never stopped, it just took different forms).

I disagree with the relationship where a service's development can only
proceed at the speed of a driver (or driver's because where there is
one, there are plenty more).

Thanks,
Anita.

That's entirely up to them, but
there's no policy or technical reason it can not.

-Jim


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Feb 24, 2015 by Anita_Kuno (21,320 points)   3 3 4
0 votes

Anita Kuno anteaya@anteaya.info writes:

I'd like to make sure that if Infra is saying that CI jobs on drivers
can gate that this is a substantial difference from what I have been
saying for a year, specifically that they can't and won't.

For the purposes of this conversation, the distinction between drivers
and any other type of project is not important. The only thing that can
not gate is a CI system other than the one we run. That's not a change.

Again, most driver CI systems are third-party systems because they must
be for technical reasons (and therefore, they can not gate for that
reason). If they can be run upstream, there's no reason they can't
gate. And there are some instances where we would be strongly advised
to -- the "default open-source" driver for a system, for instance. Most
of those happen to be in-tree at the moment.

-Jim


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Feb 24, 2015 by James_E._Blair (5,080 points)   1 6 8
0 votes

this is a reflection of the discussion I just had on #openstack-infra; it's
about (re-)using the central CI infrastructure for our Open-Source DRBD
driver too.
As this is done now, I'd like to discuss a few points.

First of all -- thanks to everybody involved; if you happen to find someone
(Europe is a bad timezone for Openstack, I guess), people on IRC are very
helpful.

Secondly,

«Yup, so the two things I will start with is that multinode testing is
still really rudimentary, we only just got tempest sort of working with
it. ... »
I didn't follow all the conversations on the ML - is there an update for
multi-node testing?
What's the policy? "Good idea, but not needed" or "A must-have"?

Especially for distributed/replicated storage drivers (like DRBD ;) I guess
it would make sense to have some.

Third point, mostly of interest for other driver maintainers: we're
following the various CI runs, and sporadically check failures to see
whether we can improve our driver or the devstack installation script.
For that we use "Kibana" on http://logstash.openstack.org, with a filter
string of

project:"openstack/cinder" AND build_status:"FAILURE" AND
build_name:"check-tempest-dsvm-full-drbd-devstack-nv" AND "Detailed logs"

This shows the initial line of the failed devstack runs (the one including
the line with the log URL), which we can then paste into a script that
fetches the (for us) relevant files for further analysis.
The newest failures are already at the top.

Another nice feature is using the existing Graphite installation to get
a visual display of the success/failure ratio over time[1]; here we can see
the impact of individual changes, eg. on June 19th we diagnosed (and fixed)
an udev/blkid race with the kernel-attach, since then the number of
failures has clearly gone down. I just pushed another patch that should us
bring even more "into the green area" ;)

One more idea is to watch http://status.openstack.org/zuul/? with a filter
string of "Cinder", and to open reviews that will finish soon, so that
"current" behaviour can be easily checked.

I hope that this helps other people a bit.

Regards,

Phil

Ad 1:
http://graphite.openstack.org/render/?width=600&height=344&_salt=1434709688.361&from=-7days&title=DRBD%20Cinder%2FDevstack%20stats&colorList=red%2Cgreen%2Cblue&target=stats_counts.zuul.pipeline.check.job.check-tempest-dsvm-full-drbd-devstack-nv.FAILURE&target=stats_counts.zuul.pipeline.check.job.check-tempest-dsvm-full-drbd-devstack-nv.SUCCESS

--
: Ing. Philipp Marek
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com :

DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jun 22, 2015 by Philipp_Marek (1,760 points)   1 4
...