settingsLogin | Registersettings

[openstack-dev] OpenStack moving both too fast and too slow at the same time

0 votes

This email is meant to be the ML discussion of a question I brought up
during the TC meeting on April 25th.; [1]

The TL;DR version is:

Reading the user survey [2], I see the same issues time and time again.
Pages 18-19 of the survey are especially common points.
Things move too fast, no LTS release, upgrades are terrifying for
anything that isn't N-1 -> N.
These come up time and time again
How is the TC working with the dev teams to address these critical issues?

I asked this because on page 18 is this comment:

"Most large customers move slowly and thus are running older versions,
which are EOL upstream sometimes before they even deploy them."

This is exactly what we're seeing with some of our customers and I
wanted to ask the TC about it.

Thanks,

-Drew

[1]
http://eavesdrop.openstack.org/meetings/tc/2017/tc.2017-04-25-20.00.log.html#l-177
[2] https://www.openstack.org/assets/survey/April2017SurveyReport.pdf


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked May 10, 2017 in openstack-dev by Drew_Fisher (600 points)   1 3

72 Responses

0 votes

On 05/03/2017 02:00 PM, Drew Fisher wrote:

Reading the user survey [2], I see the same issues time and time again.
Pages 18-19 of the survey are especially common points.
Things move too fast, no LTS release, upgrades are terrifying for
anything that isn't N-1 -> N.
These come up time and time again
How is the TC working with the dev teams to address these critical issues?

With my upstream nova developer hat on, it'd be painful to try to enable
upgrades for anything other than N-1 -> N. (Heck, it's sometimes painful to
even ensure that N-1 -> N works properly.)

The team I work for made the decision to skip Liberty and upgrade directly from
Kilo to Mitaka. We made it work, but the RPC and object versioning issues were
a bit finicky.

I asked this because on page 18 is this comment:

"Most large customers move slowly and thus are running older versions,
which are EOL upstream sometimes before they even deploy them."

This is exactly what we're seeing with some of our customers and I
wanted to ask the TC about it.

Us too. I'm not sure there is a simple solution. To some extent I suppose
that's what distro folks get paid for...to do stuff that upstream can't (or
won't) do.

Chris


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 4, 2017 by Chris_Friesen (20,420 points)   3 16 24
0 votes

On 05/03/2017 09:30 PM, Chris Friesen wrote:
On 05/03/2017 02:00 PM, Drew Fisher wrote:

I asked this because on page 18 is this comment:

"Most large customers move slowly and thus are running older versions,
which are EOL upstream sometimes before they even deploy them."

This is exactly what we're seeing with some of our customers and I
wanted to ask the TC about it.

Us too. I'm not sure there is a simple solution. To some extent I
suppose that's what distro folks get paid for...to do stuff that
upstream can't (or won't) do.

Chris

Us distro folks don't like it either :P

--
Matthew Thode (prometheanfire)


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

responded May 4, 2017 by prometheanfire_at_ge (6,880 points)   1 4 5
0 votes

On Wed, 3 May 2017, Drew Fisher wrote:

This email is meant to be the ML discussion of a question I brought up
during the TC meeting on April 25th.; [1]

Thanks for starting this Drew, I hope my mentioning it in my tc
report email wasn't too much of a nag.

I've added [tc] and [all] tags to the subject in case people are
filtering. More within.

The TL;DR version is:

Reading the user survey [2], I see the same issues time and time again.
Pages 18-19 of the survey are especially common points.
Things move too fast, no LTS release, upgrades are terrifying for
anything that isn't N-1 -> N.
These come up time and time again
How is the TC working with the dev teams to address these critical issues?

As I recall the "OpenStack-wide Goals"a are supposed to help address
some of this sort of thing but it of course relies on people first
proposing and detailing goals and then there actually being people
to act on them. The first part was happening atb but it's not
clear if that's the current way.

Having people is the hard part. Given the current contribution
model[c] that pretty much means enterprises ponying up the people do
the work. If they don't do that then the work won't get done, and
people won't buy the products they are supporting, I guess? Seems a
sad state of affairs.

There's also an issue where we seem to have decided that it is only
appropriate to demand a very small number of goals per cycle
(because each project already has too much on their plate, or too big
a backlog, relative to resources). It might be that as the
Technical Committe is could be legitimate to make a larger demand.
(Or it could be completely crazy.)

I asked this because on page 18 is this comment:

"Most large customers move slowly and thus are running older versions,
which are EOL upstream sometimes before they even deploy them."

Can someone with more of the history give more detail on where the
expectation arose that upstream ought to be responsible things like
long term support? I had always understood that such features were
part of the way in which the corporately avaialable products added
value?

This is exactly what we're seeing with some of our customers and I
wanted to ask the TC about it.

I know you're not speaking as the voice of your employer when making
this message, so this is not directed at you, but from what I can
tell Oracle's presense upstream (both reviews and commits) in Ocata
and thus far in Pike has not been huge. Maybe that's something that
needs to change to keep the customers happy? Or at all.

[c]: There's talk that the current model will change from devs hired
to do OpenStack development being the main engine of contribution to
users of OpenStack, who happen to be devs, being the main engine. Do
we know the slope on that trend?

--
Chris Dent ┬──┬◡ノ(° -°ノ) https://anticdent.org/
freenode: cdent tw: @anticdent__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

responded May 4, 2017 by cdent_plus_os_at_ant (12,800 points)   2 2 6
0 votes

Chris Dent wrote:
On Wed, 3 May 2017, Drew Fisher wrote:

"Most large customers move slowly and thus are running older versions,
which are EOL upstream sometimes before they even deploy them."

Can someone with more of the history give more detail on where the
expectation arose that upstream ought to be responsible things like
long term support? I had always understood that such features were
part of the way in which the corporately avaialable products added
value?

We started with no stable branches, we were just producing releases and
ensuring that updates vaguely worked from N-1 to N. There were a lot of
distributions, and they all maintained their own stable branches,
handling backport of critical fixes. That is a pretty classic upstream /
downstream model.

Some of us (including me) spotted the obvious duplication of effort
there, and encouraged distributions to share that stable branch
maintenance work rather than duplicate it. Here the stable branches were
born, mostly through a collaboration between Red Hat developers and
Canonical developers. All was well. Nobody was saying LTS back then
because OpenStack was barely usable so nobody wanted to stay on any
given version for too long.

Maintaining stable branches has a cost. Keeping the infrastructure that
ensures that stable branches are actually working is a complex endeavor
that requires people to constantly pay attention. As time passed, we saw
the involvement of distro packagers become more limited. We therefore
limited the number of stable branches (and the length of time we
maintained them) to match the staffing of that team. Fast-forward to
today: the stable team is mostly one person, who is now out of his job
and seeking employment.

In parallel, OpenStack became more stable, so the demand for longer-term
maintenance is stronger. People still expect "upstream" to provide it,
not realizing upstream is made of people employed by various
organizations, and that apparently their interest in funding work in
that area is pretty dead.

I agree that our current stable branch model is inappropriate:
maintaining stable branches for one year only is a bit useless. But I
only see two outcomes:

1/ The OpenStack community still thinks there is a lot of value in doing
this work upstream, in which case organizations should invest resources
in making that happen (starting with giving the Stable branch
maintenance PTL a job), and then, yes, we should definitely consider
things like LTS or longer periods of support for stable branches, to
match the evolving usage of OpenStack.

2/ The OpenStack community thinks this is better handled downstream, and
we should just get rid of them completely. This is a valid approach, and
a lot of other open source communities just do that.

The current reality in terms of invested resources points to (2). I
personally would prefer (1), because that lets us address security
issues more efficiently and avoids duplicating effort downstream. But
unfortunately I don't control where development resources are posted.

--
Thierry Carrez (ttx)


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 4, 2017 by Thierry_Carrez (57,480 points)   3 8 13
0 votes

Excerpts from Drew Fisher's message of 2017-05-03 14:00:53 -0600:

This email is meant to be the ML discussion of a question I brought up
during the TC meeting on April 25th.; [1]

Thanks for starting this thread, Drew. I'll try to respond, but I
know a lot of folks are preparing for the summit next week, so it
may be a little quiet around here until after everyone is home.

The TL;DR version is:

Reading the user survey [2], I see the same issues time and time again.
Pages 18-19 of the survey are especially common points.

I was also interested in those comments and noticed that, as you
say, some are recurring themes. That reinforces in my mind that we
haven't adequately communicated the background behind some decisions
we've made in the past, or what we would need to do to make progress
on stalled initiatives. I've started trying to address some of
those issues [1], and I'll be continuing that work after the summit.

[1] https://doughellmann.com/blog/2017/04/20/lessons-learned-from-working-on-large-scale-cross-project-initiatives-in-openstack/

Things move too fast,

I have to say, after so many years of hearing that we weren't moving
fast enough this one was a big surprise. :-) I'm not sure if that's
good or bad, or if it just means we now have a completely different
set of people responding to the user survey.

no LTS release,

Over the past couple of years we have shifted the majority of the
backport review work off of a centralized team so that the individual
project teams are responsible for establishing their own stable
review groups. We've also changed the way we handle stable releases,
so that we now encourage projects to tag a release when they need
it instead of waiting and trying to tag all of the projects together
at the same time. As a result of these changes, we've been seeing
more stable releases for the branches we do maintain, giving users
more actual bug fix releases for those series.

That said, there are two main reasons we are unlikely to add more
stable releases or maintain any releases for longer: we need more
people to do the work, and we need to find a way to do that work
that doesn't hurt our ability to work on master.

We do still have a stable team responsible for ensuring that projects
are following the policies for stable releases, and that team needs
more participation. I'm sure the project teams would appreciate
having more help with backports and reviews on their stable branches,
too. Getting contributors to work on those tasks has been difficult
since the very beginning of the project.

It has been difficult to attract contributors to this area in part
due to the scope of work that is necessary to say that the community
supports those releases. We need the older versions of the deployment
platforms available in our CI systems to run the automated tests.
We need supported versions of the development tools (setuptools and
pip are especially problemmatic). We need supported versions of
the various libraries and system-level dependencies like libvirt.
I'm sure the stable maintenance team could add to that list, but
the point is that it's not just a matter of saying we want to do
it, or even that we will do it.

upgrades are terrifying for anything that isn't N-1 -> N.

The OpenStack community has a strong culture of testing. We have
reasonable testing in place to balance our ability to ensure that
N-1 -> N upgrades work and as a result upgrades are easier than
ever. It seems quite a few users are still on the older versions
of the software that don't have some of those improvements. It's
not the ideal answer, but their experience will continue to improve
as they move forward onto newer releases.

Meanwhile, adding more combinations of upgrades to handle N-M -> N
changes our ability to simplify the applications by removing technical
debt and by deprecating configuration options (reducing complexity
by cutting the number of configuration options has also been a
long-standing request from users). It also means more people are
needed to keep those older releases running in CI, so that the
upgrade jobs are reliable (see the discussion above about why that
is an issue).

These come up time and time again
How is the TC working with the dev teams to address these critical issues?

I asked this because on page 18 is this comment:

"Most large customers move slowly and thus are running older versions,
which are EOL upstream sometimes before they even deploy them."

This is exactly what we're seeing with some of our customers and I
wanted to ask the TC about it.

The contributors to OpenStack are not a free labor pool for the
consumers of the project. Just like with any other open source
project, the work is done by the people who show up, and we're all
motivated to work on different things. Many (most?) of us are paid
by companies selling products or services based on OpenStack. Those
companies apply resources, in the form of contributor time and
attention, based on their users' needs. The TC and other community
leaders have tried to respond to that particular source of contributor
motivation while still honoring contributions of all forms by
organizing the project to enable all contributors to achieve their
goals, regardless of whether they are working for a "vendor" of
some sort, or are a user building their own packages and relying
on the community for support. Having done both while I've been
active with OpenStack, I can honestly say that the best way to get
what you want is to participate in creating it. If there were
people trying to do this work, I expect that we would find a way
to make it possible.

For anyone interested in contributing to stable release maintenance,
the best course of action is to get involved with the team responsible
for organizing that work to learn our current processes and policies.
When that team grows large enough that it's clear we can sustain
more stable releases or upgrade combinations, the people involved
will have a clear enough understanding of the problem space that
they will be able to design the necessary policy and process changes.

Doug

Thanks,

-Drew

[1]
http://eavesdrop.openstack.org/meetings/tc/2017/tc.2017-04-25-20.00.log.html#l-177
[2] https://www.openstack.org/assets/survey/April2017SurveyReport.pdf


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 4, 2017 by Doug_Hellmann (87,520 points)   3 4 10
0 votes

On Thu, May 4, 2017 at 5:32 AM, Chris Dent cdent+os@anticdent.org wrote:
On Wed, 3 May 2017, Drew Fisher wrote:

This email is meant to be the ML discussion of a question I brought up
during the TC meeting on April 25th.; [1]

Thanks for starting this Drew, I hope my mentioning it in my tc
report email wasn't too much of a nag.

I've added [tc] and [all] tags to the subject in case people are
filtering. More within.

The TL;DR version is:

Reading the user survey [2], I see the same issues time and time again.
Pages 18-19 of the survey are especially common points.
Things move too fast, no LTS release, upgrades are terrifying for
anything that isn't N-1 -> N.
These come up time and time again
How is the TC working with the dev teams to address these critical issues?

As I recall the "OpenStack-wide Goals"[a] are supposed to help address
some of this sort of thing but it of course relies on people first
proposing and detailing goals and then there actually being people
to act on them. The first part was happening at[b] but it's not
clear if that's the current way.

Having people is the hard part. Given the current contribution
model[c] that pretty much means enterprises ponying up the people do
the work. If they don't do that then the work won't get done, and
people won't buy the products they are supporting, I guess? Seems a
sad state of affairs.

There's also an issue where we seem to have decided that it is only
appropriate to demand a very small number of goals per cycle
(because each project already has too much on their plate, or too big
a backlog, relative to resources). It might be that as the
Technical Committe is could be legitimate to make a larger demand.
(Or it could be completely crazy.)

I asked this because on page 18 is this comment:

"Most large customers move slowly and thus are running older versions,
which are EOL upstream sometimes before they even deploy them."

Can someone with more of the history give more detail on where the
expectation arose that upstream ought to be responsible things like
long term support? I had always understood that such features were
part of the way in which the corporately avaialable products added
value?

This is exactly what we're seeing with some of our customers and I
wanted to ask the TC about it.

I know you're not speaking as the voice of your employer when making
this message, so this is not directed at you, but from what I can
tell Oracle's presense upstream (both reviews and commits) in Ocata
and thus far in Pike has not been huge. Maybe that's something that
needs to change to keep the customers happy? Or at all.

Probably because they are still on Kilo. Not sure how much they could
be contributing to the current when their customers are demanding that
something is rock solid which by now looks nothing like the current
upstream. I think this is part of the problem as the upstream can
tend to outpace anyone else in terms of features or anything else. I
think the the bigger question could be what's the benefit of
continuing to press forward and add yet more features when consumers
cannot keep up to consume these? Personally I think usability (and
some stability) sometimes tends to take a backseat to features in the
upstream which is unfortunate because it makes these problems worse.

Thanks,
-Alex

[a]: https://governance.openstack.org/tc/goals/index.html
[b]: https://etherpad.openstack.org/p/community-goals
[c]: There's talk that the current model will change from devs hired
to do OpenStack development being the main engine of contribution to
users of OpenStack, who happen to be devs, being the main engine. Do
we know the slope on that trend?

--
Chris Dent ┬──┬◡ノ(° -°ノ) https://anticdent.org/
freenode: cdent tw: @anticdent


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 4, 2017 by aschultz_at_redhat.c (5,800 points)   2 2 4
0 votes

On Thu, May 04, 2017 at 04:14:07PM +0200, Thierry Carrez wrote:
:Chris Dent wrote:
:> On Wed, 3 May 2017, Drew Fisher wrote:
:>> "Most large customers move slowly and thus are running older versions,
:>> which are EOL upstream sometimes before they even deploy them."
:>
:> Can someone with more of the history give more detail on where the
:> expectation arose that upstream ought to be responsible things like
:> long term support? I had always understood that such features were
:> part of the way in which the corporately avaialable products added
:> value?

:In parallel, OpenStack became more stable, so the demand for longer-term
:maintenance is stronger. People still expect "upstream" to provide it,
:not realizing upstream is made of people employed by various
:organizations, and that apparently their interest in funding work in
:that area is pretty dead.

Wearing my Operator hat I don't really care if "LTS" comes from
upstream or downstream. I think the upstream expectation has
developed becuase there has been some upstream efforts and as far as I
can see no recent downstream efforts in support of stable releases,
though obviously I mostly pay attention to "my" distro so may be
missing things in this space.

Having watched this for some time I agree with everything Thierry has
said.

The increasing demand for "LTS" like releases is definitely a tribute
to the overall maturity of core services. I used to be desperate for
the next release and back porting patches into custom packages just to
keep things working.

Now if I belived Ubuntu (which my world OpenStack and otherwise
happens to be built on) would provide a direct upgrade path from their
16.04 released OpenStack to what ever lands in their next LTS I'd
probably sit rather happily on that. Which is a hugely positive shift.

:I agree that our current stable branch model is inappropriate:
:maintaining stable branches for one year only is a bit useless. But I
:only see two outcomes:
:
:1/ The OpenStack community still thinks there is a lot of value in doing
:this work upstream, in which case organizations should invest resources
:in making that happen (starting with giving the Stable branch
:maintenance PTL a job), and then, yes, we should definitely consider
:things like LTS or longer periods of support for stable branches, to
:match the evolving usage of OpenStack.
:
:2/ The OpenStack community thinks this is better handled downstream, and
:we should just get rid of them completely. This is a valid approach, and
:a lot of other open source communities just do that.
:
:The current reality in terms of invested resources points to (2). I
:personally would prefer (1), because that lets us address security
:issues more efficiently and avoids duplicating effort downstream. But
:unfortunately I don't control where development resources are posted.

Yes it seems that way to me as well.

just killing the stable branch model without some plan either
internally or externally to provide a better stability story seems
like it would send the wrong signal. So I'd much prefer the distro
people to either back option 1) with significant resources so it can
really work or make public commitments to handle option 2) in a
reasonable way.

-Jon


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 4, 2017 by jon_at_csail.mit.edu (4,720 points)   1 4 7
0 votes

On 05/04/2017 10:57 AM, Doug Hellmann wrote:
Excerpts from Drew Fisher's message of 2017-05-03 14:00:53 -0600:

These come up time and time again
How is the TC working with the dev teams to address these critical issues?

I asked this because on page 18 is this comment:

"Most large customers move slowly and thus are running older versions,
which are EOL upstream sometimes before they even deploy them."

This is exactly what we're seeing with some of our customers and I
wanted to ask the TC about it.

The contributors to OpenStack are not a free labor pool for the
consumers of the project.

1000 times THIS.

You generally get out of the open source projects what you put into them
-- either time, money, or both.

Best,
-jay


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 4, 2017 by Jay_Pipes (59,760 points)   3 11 14
0 votes

On 04/05/17 11:18 -0400, Jonathan Proulx wrote:
On Thu, May 04, 2017 at 04:14:07PM +0200, Thierry Carrez wrote:
:I agree that our current stable branch model is inappropriate:
:maintaining stable branches for one year only is a bit useless. But I
:only see two outcomes:
:
:1/ The OpenStack community still thinks there is a lot of value in doing
:this work upstream, in which case organizations should invest resources
:in making that happen (starting with giving the Stable branch
:maintenance PTL a job), and then, yes, we should definitely consider
:things like LTS or longer periods of support for stable branches, to
:match the evolving usage of OpenStack.
:
:2/ The OpenStack community thinks this is better handled downstream, and
:we should just get rid of them completely. This is a valid approach, and
:a lot of other open source communities just do that.
:
:The current reality in terms of invested resources points to (2). I
:personally would prefer (1), because that lets us address security
:issues more efficiently and avoids duplicating effort downstream. But
:unfortunately I don't control where development resources are posted.

Have there been issues with downstream distros not addressing security fixes
properly?

Yes it seems that way to me as well.

just killing the stable branch model without some plan either
internally or externally to provide a better stability story seems
like it would send the wrong signal. So I'd much prefer the distro
people to either back option 1) with significant resources so it can
really work or make public commitments to handle option 2) in a
reasonable way.

I think downstream distros are already doing #2, unless I'm missing something.
How public/vocal they are about it might be a different discussion.

I'd prefer #1 too because I'd rather have everything upstream. However, with the
current flux of people, the current roadmaps and the current status of the
community, it's unrealistic for us to expect #1 to happen. So, I'd rather
dedicate time documenting/communicating #2 properly.

Now, one big problem with LTS releases of OpenStack (regardless they happen
upstream or downstream) is the upgrade path, which is one of the problems Drew
raised.

--
@flaper87
Flavio Percoco


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

responded May 4, 2017 by Flavio_Percoco (36,960 points)   3 7 11
0 votes

Flavio Percoco wrote:
On 04/05/17 11:18 -0400, Jonathan Proulx wrote:

On Thu, May 04, 2017 at 04:14:07PM +0200, Thierry Carrez wrote:
:I agree that our current stable branch model is inappropriate:
:maintaining stable branches for one year only is a bit useless. But I
:only see two outcomes:
:
:1/ The OpenStack community still thinks there is a lot of value in doing
:this work upstream, in which case organizations should invest resources
:in making that happen (starting with giving the Stable branch
:maintenance PTL a job), and then, yes, we should definitely consider
:things like LTS or longer periods of support for stable branches, to
:match the evolving usage of OpenStack.
:
:2/ The OpenStack community thinks this is better handled downstream, and
:we should just get rid of them completely. This is a valid approach, and
:a lot of other open source communities just do that.
:
:The current reality in terms of invested resources points to (2). I
:personally would prefer (1), because that lets us address security
:issues more efficiently and avoids duplicating effort downstream. But
:unfortunately I don't control where development resources are posted.

Have there been issues with downstream distros not addressing security
fixes properly?

No, not at all -- but usually they package upstream vulnerability fixes,
which are produced on stable branches. In mode #2 we would only patch
master, forcing downstream to do backports for more branches. That is
what I meant by "more efficiently".

Sorry for being unclear.

--
Thierry Carrez (ttx)


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 4, 2017 by Thierry_Carrez (57,480 points)   3 8 13
...