settingsLogin | Registersettings

[openstack-dev] [all][tc] Languages vs. Scope of "OpenStack"

0 votes

Hi everyone,

The discussion on the addition of golang focuses on estimating community
costs vs. technical benefits, so that the TC can make the right call for
"OpenStack". From that discussion, we established that the costs for
cross-project teams (Infra...) seem to be workable. There is still
significant community fragmentation effects as we add another language,
creating language-expert silos, duplicating efforts, and losing some
productivity as some needlessly rewrite things in golang. But we seem
generally ready to accept that cost because we are missing a tool in our
toolbelt: a language that lets us do such native optimization.

We have a number of projects in OpenStack that are more low-level than
others and which require such optimization, and for those projects (or
subprojects) our current mix of language is not satisfactory. The Swift
team in particular has spent a lot of time trying to get the I/O
behavior they require with hard disk access using Python, and they
certainly did not jump on the golang bandwagon lightly.

I believe the programming languages you need in OpenStack official
projects are a function of the scope you define for OpenStack official
projects. We wouldn't need to have JavaScript in the mix if we
considered that web interfaces that purely consume OpenStack APIs are
projects that consume OpenStack, rather than part of OpenStack itself.

In the same vein, if we consider lower-level projects (which often
require such native optimization) as part of "OpenStack", rather than as
external open source projects that should be integrated by "OpenStack",
then we need a language like golang in our toolbelt. There is basically
no point in saying no to golang in OpenStack if we need lower-level
native optimization in OpenStack: we'll have to accept the community
cost that comes with such a community scope.

So the real question we need to answer is... where does OpenStack stop,
and where does the wider open source community start ? If OpenStack is
purely an "integration engine", glue code for other lower-level
technologies like hypervisors, databases, or distributed block storage,
then the scope is limited, Python should be plenty enough, and we don't
need to fragment our community. If OpenStack is "whatever it takes to
reach our mission", then yes we need to add one language to cover
lower-level/native optimization, because we'll need that... and we need
to accept the community cost as a consequence of that scope choice.
Those are the only two options on the table.

I'm actually not sure what is the best answer. But I'm convinced we, as
a community, need to have a clear answer to that. We've been avoiding
that clear answer until now, creating tension between the advocates of
an ASF-like collection of tools and the advocates of a
tighter-integrated "openstack" product. We have created silos and
specialized areas as we got into the business of developing time-series
databases or SDNs. As a result, it's not "one community" anymore. Should
we further encourage that, or should we focus on what the core of our
mission is, what we have in common, this integration engine that binds
all those other open source projects into one programmable
infrastructure solution ?

--
Thierry Carrez (ttx)


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked May 19, 2016 in openstack-dev by Thierry_Carrez (57,480 points)   3 8 13
retagged Jan 25, 2017 by admin

57 Responses

0 votes

On Thu, May 19, 2016 at 6:19 AM, Thierry Carrez thierry@openstack.org
wrote:

Hi everyone,

The discussion on the addition of golang focuses on estimating community
costs vs. technical benefits, so that the TC can make the right call for
"OpenStack". From that discussion, we established that the costs for
cross-project teams (Infra...) seem to be workable. There is still
significant community fragmentation effects as we add another language,
creating language-expert silos, duplicating efforts, and losing some
productivity as some needlessly rewrite things in golang. But we seem
generally ready to accept that cost because we are missing a tool in our
toolbelt: a language that lets us do such native optimization.

We have a number of projects in OpenStack that are more low-level than
others and which require such optimization, and for those projects (or
subprojects) our current mix of language is not satisfactory. The Swift
team in particular has spent a lot of time trying to get the I/O behavior
they require with hard disk access using Python, and they certainly did not
jump on the golang bandwagon lightly.

I agree with this. it is totally reasonable to add golang (or a tool to
fill the need for native optimizations). The fragmentation concerns are
real, but filling this gap in the openstack tool-chain is important. It
will garner more interest in the project as it opens the doors to examine
the native level optimizations that are exceptionally hard in Python. While
this all could be done in Python, I would argue that there are legitimate
cases where a tool like golang is going to be a "better" fit for the job. I
would also argue that the TC should provide a strong guidance on where
golang should be used (initially). Specifically that it should be used in
cases where Python can demonstrably be shown to be the inferior tool for
the job and not as whole-sale replacement(s) for the core API/end-user
interactions. The type of break that the swift team is advocating for is a
great starting place. With the acknowledgement from the -infra team that
this is reasonable and workable, I am willing (with my TC hat on), agree to
include golang provided the recommendations are outlined for the initial
use-cases. As we see where things shake out with these initial use-cases I
am sure we will see expanded uses of golang that make sense. I do know that
policing such uses is difficult at best, and I don't think it is something
that should be done via technology but more of a social contract and
documented as the recommended for initial cases of golang (or other similar
tool-chain) inclusion.

I believe the programming languages you need in OpenStack official
projects are a function of the scope you define for OpenStack official
projects. We wouldn't need to have JavaScript in the mix if we considered
that web interfaces that purely consume OpenStack APIs are projects that
consume OpenStack, rather than part of OpenStack itself.

In the same vein, if we consider lower-level projects (which often require
such native optimization) as part of "OpenStack", rather than as external
open source projects that should be integrated by "OpenStack", then we need
a language like golang in our toolbelt. There is basically no point in
saying no to golang in OpenStack if we need lower-level native optimization
in OpenStack: we'll have to accept the community cost that comes with such
a community scope.

So the real question we need to answer is... where does OpenStack stop,
and where does the wider open source community start ? If OpenStack is
purely an "integration engine", glue code for other lower-level
technologies like hypervisors, databases, or distributed block storage,
then the scope is limited, Python should be plenty enough, and we don't
need to fragment our community. If OpenStack is "whatever it takes to reach
our mission", then yes we need to add one language to cover
lower-level/native optimization, because we'll need that... and we need to
accept the community cost as a consequence of that scope choice. Those are
the only two options on the table.

It would be disingenuous to say we are a pure integration engine. If swift
or a similar project is outside the scope of openstack, frankly, I view
Keystone as outside of the scope of openstack. Keystone (while in a
different position due to a lot of heavy requirements for auth in
openstack) is not a simple "integrate an underlying technology with other
things with some tracking of the resources". Keystone provides a real added
value, while perhaps not as low-level as swift, that extends significantly
beyond the scope of "glue" of it's individual components. I do believe the
lower level technologies have a place in "openstack's big tent" but should
be evaluated carefully before inclusion; I am not advocating we include
MySQL, PGSQL, or RabbitMQ as "part of the big tent". I think object storage
is an important thing within the cloud ecosystem, and by it's nature swift
belongs as part of the big tent.

The cost for golang, in my view after having a large number of
conversations, is worth it. This conversation will continue to come up in
different forms if we nix golang here, the low level optimizations are
relevant and the projects (initially swift) that are running up against
these issues are in-fact a part of OpenStack today.. I also want to make
sure we, as the TC and community, are not evaluating a specific project
here, but OpenStack as a whole. If we are evaluating one project
specifically and making a change based upon where it is in the ecosystem,
we should step way back and evaluate why this just now has become part of
the conversation and address that issue first

I'm actually not sure what is the best answer. But I'm convinced we, as a
community, need to have a clear answer to that. We've been avoiding that
clear answer until now, creating tension between the advocates of an
ASF-like collection of tools and the advocates of a tighter-integrated
"openstack" product. We have created silos and specialized areas as we got
into the business of developing time-series databases or SDNs. As a result,
it's not "one community" anymore. Should we further encourage that, or
should we focus on what the core of our mission is, what we have in common,
this integration engine that binds all those other open source projects
into one programmable infrastructure solution ?

--
Thierry Carrez (ttx)

I think we would do better as an ASF-like organization with a bias towards
programmable infrastructure. I do not think we should include more
time-series databases, RDBMSs, or even Message Brokers simply for the sake
of inclusion (there are examples of these in the big tent already). But we
should be careful about excluding and eliminating projects and scope as
well. We should include golang, and work on the other issues separately
(this sounds like something we should be working on setting proper
guideposts for the community, how we do that, etc, and revolves around
improving how the TC provides leadership). As part of improving the
leadership of OpenStack, we need to also work to have a clear
product-vision (and I do not mean "product" as in something specifically
sell-able). I think part of our issue and what is driving these
conversations is a lack of clear product vision which is part of the TC
providing the guideposts and improving leadership within OpenStack.

--
Morgan Fainberg (notmorgan)


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 19, 2016 by Morgan_Fainberg (17,320 points)   2 4 9
0 votes

summary:
* Defining the scope of OpenStack projects DOES NOT define the
languages needed to implement them. The considerations are
orthogonal.
* We've already defined OpenStack--it's whatever it takes to
fulfill its mission statement.

On 19 May 2016, at 6:19, Thierry Carrez wrote:

Hi everyone,

The discussion on the addition of golang focuses on estimating
community costs vs. technical benefits, so that the TC can make the
right call for "OpenStack". From that discussion, we established
that the costs for cross-project teams (Infra...) seem to be
workable. There is still significant community fragmentation
effects as we add another language, creating language-expert silos,
duplicating efforts, and losing some productivity as some
needlessly rewrite things in golang. But we seem generally ready to
accept that cost because we are missing a tool in our toolbelt: a
language that lets us do such native optimization.

We have a number of projects in OpenStack that are more low-level
than others and which require such optimization, and for those
projects (or subprojects) our current mix of language is not
satisfactory. The Swift team in particular has spent a lot of time
trying to get the I/O behavior they require with hard disk access
using Python, and they certainly did not jump on the golang
bandwagon lightly.

I believe the programming languages you need in OpenStack official
projects are a function of the scope you define for OpenStack
official projects. We wouldn't need to have JavaScript in the mix if
we considered that web interfaces that purely consume OpenStack
APIs are projects that consume OpenStack, rather than part of
OpenStack itself.

In the same vein, if we consider lower-level projects (which often
require such native optimization) as part of "OpenStack", rather
than as external open source projects that should be integrated by
"OpenStack", then we need a language like golang in our toolbelt.
There is basically no point in saying no to golang in OpenStack if
we need lower-level native optimization in OpenStack: we'll have to
accept the community cost that comes with such a community scope.

Defining "lower-level" is very hard. Since the Nova API[1] is
listening to a public network interface and coordinating with various
services in a cluster, is it low-level enough to need to consider
optimizations? Does the Nova API require optimization to handle a very
large number of connections using all of the hardware available on a
single server? If Nova is going to eek out every drop of performance
possible from a server, it probably does need to consider all kinds of
"low-level" optimizations.[2]

Because deployers of any OpenStack project do want efficient software
that is performant, all parts of OpenStack need to consider what it
takes to meet the performance demanded. Most of the time this will not
require rewriting code in a different language (that's almost never
the right answer), but my point is that I believe you're wrong that
defining the scope of OpenStack projects also defines the languages
needed to implement them. The considerations are orthogonal.

[1] substitute your favorite OpenStack project

[2] http://highscalability.com/blog/2013/5/13/the-secret-to-10-million-concurrent-connections-the-kernel-i.html

So the real question we need to answer is... where does OpenStack
stop, and where does the wider open source community start ? If
OpenStack is purely an "integration engine", glue code for other
lower-level technologies like hypervisors, databases, or distributed
block storage, then the scope is limited, Python should be plenty
enough, and we don't need to fragment our community. If OpenStack is
"whatever it takes to reach our mission", then yes we need to add one
language to cover lower-level/native optimization, because we'll
need that... and we need to accept the community cost as a
consequence of that scope choice. Those are the only two options on
the table.

I'm actually not sure what is the best answer. But I'm convinced we,
as a community, need to have a clear answer to that. We've been
avoiding that clear answer until now, creating tension between the
advocates of an ASF-like collection of tools and the advocates of a
tighter-integrated "openstack" product. We have created silos and
specialized areas as we got into the business of developing time-
series databases or SDNs. As a result, it's not "one community"
anymore. Should we further encourage that, or should we focus on
what the core of our mission is, what we have in common, this
integration engine that binds all those other open source projects
into one programmable infrastructure solution ?

You said the answer in your question. OpenStack isn't defined as an
integration engine[3]. The definition of OpenStack is whatever it
takes to fulfill our mission[4][5]. I don't mean that as a tautology.
I mean that we've already gone to the effort of defining OpenStack. It's
our mission statement. We're all about building a cloud platform upon
which people can run their apps ("cloud-native" or otherwise), so we
write the software needed to do that.

So where does OpenStack stop and the wider community start? OpenStack
includes the projects needed to fulfill its mission.

[3] There are foundation staff who have used that phrase to help
describe some of the work we do (mostly to explain how OpenStack still
matters in a docker/kubernetes/container world). While that may be good for
explaining and bringing people in, it's not the definition of
OpenStack.

[4] "To produce a ubiquitous Open Source Cloud Computing platform that
is easy to use, simple to implement, interoperable between
deployments, works well at all scales, and meets the needs of users
and operators of both public and private clouds."

[5] From openstack.org: "OpenStack software controls large pools of
compute, storage, and networking resources throughout a datacenter,
managed through a dashboard or via the OpenStack API."


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

responded May 19, 2016 by John_Dickinson (10,400 points)   2 4 7
0 votes

Morgan Fainberg wrote:

On Thu, May 19, 2016 at 6:19 AM, Thierry Carrez <thierry@openstack.org
thierry@openstack.org> wrote:

Hi everyone,

The discussion on the addition of golang focuses on estimating
community costs vs. technical benefits, so that the TC can make the
right call for "OpenStack". From that discussion, we established
that the costs for cross-project teams (Infra...) seem to be
workable. There is still significant community fragmentation effects
as we add another language, creating language-expert silos,
duplicating efforts, and losing some productivity as some needlessly
rewrite things in golang. But we seem generally ready to accept that
cost because we are missing a tool in our toolbelt: a language that
lets us do such native optimization.

We have a number of projects in OpenStack that are more low-level
than others and which require such optimization, and for those
projects (or subprojects) our current mix of language is not
satisfactory. The Swift team in particular has spent a lot of time
trying to get the I/O behavior they require with hard disk access
using Python, and they certainly did not jump on the golang
bandwagon lightly.

I agree with this. it is totally reasonable to add golang (or a tool to
fill the need for native optimizations). The fragmentation concerns are
real, but filling this gap in the openstack tool-chain is important. It
will garner more interest in the project as it opens the doors to
examine the native level optimizations that are exceptionally hard in
Python. While this all could be done in Python, I would argue that there
are legitimate cases where a tool like golang is going to be a "better"
fit for the job. I would also argue that the TC should provide a strong
guidance on where golang should be used (initially). Specifically that
it should be used in cases where Python can demonstrably be shown to be
the inferior tool for the job and not as whole-sale replacement(s) for
the core API/end-user interactions. The type of break that the swift
team is advocating for is a great starting place. With the
acknowledgement from the -infra team that this is reasonable and
workable, I am willing (with my TC hat on), agree to include golang
provided the recommendations are outlined for the initial use-cases. As
we see where things shake out with these initial use-cases I am sure we
will see expanded uses of golang that make sense. I do know that
policing such uses is difficult at best, and I don't think it is
something that should be done via technology but more of a social
contract and documented as the recommended for initial cases of golang
(or other similar tool-chain) inclusion.

I believe the programming languages you need in OpenStack official
projects are a function of the scope you define for OpenStack
official projects. We wouldn't need to have JavaScript in the mix if
we considered that web interfaces that purely consume OpenStack APIs
are projects that consume OpenStack, rather than part of OpenStack
itself.

In the same vein, if we consider lower-level projects (which often
require such native optimization) as part of "OpenStack", rather
than as external open source projects that should be integrated by
"OpenStack", then we need a language like golang in our toolbelt.
There is basically no point in saying no to golang in OpenStack if
we need lower-level native optimization in OpenStack: we'll have to
accept the community cost that comes with such a community scope.

So the real question we need to answer is... where does OpenStack
stop, and where does the wider open source community start ? If
OpenStack is purely an "integration engine", glue code for other
lower-level technologies like hypervisors, databases, or distributed
block storage, then the scope is limited, Python should be plenty
enough, and we don't need to fragment our community. If OpenStack is
"whatever it takes to reach our mission", then yes we need to add
one language to cover lower-level/native optimization, because we'll
need that... and we need to accept the community cost as a
consequence of that scope choice. Those are the only two options on
the table.

It would be disingenuous to say we are a pure integration engine. If
swift or a similar project is outside the scope of openstack, frankly, I
view Keystone as outside of the scope of openstack. Keystone (while in a
different position due to a lot of heavy requirements for auth in
openstack) is not a simple "integrate an underlying technology with
other things with some tracking of the resources". Keystone provides a
real added value, while perhaps not as low-level as swift, that extends
significantly beyond the scope of "glue" of it's individual components.
I do believe the lower level technologies have a place in "openstack's
big tent" but should be evaluated carefully before inclusion; I am not
advocating we include MySQL, PGSQL, or RabbitMQ as "part of the big
tent". I think object storage is an important thing within the cloud
ecosystem, and by it's nature swift belongs as part of the big tent.
The cost for golang, in my view after having a large number of
conversations, is worth it. This conversation will continue to come up
in different forms if we nix golang here, the low level optimizations
are relevant and the projects (initially swift) that are running up
against these issues are in-fact a part of OpenStack today.. I also want
to make sure we, as the TC and community, are not evaluating a specific
project here, but OpenStack as a whole. If we are evaluating one project
specifically and making a change based upon where it is in the
ecosystem, we should step way back and evaluate why this just now has
become part of the conversation and address that issue first

+1

I really dislike the words 'integration engine' because IMHO its not
honest, its not interesting and I don't feel its really innovative and
attractive to developers that are coming to work on a opensource project
(for say there first time). If I was a new developer out of college and
was going to be working on a 'integration engine' (and/or glue engine)
I'd be running for the hills. We really need to work on a better terms
here IMHO ;)

I'm actually not sure what is the best answer. But I'm convinced we,
as a community, need to have a clear answer to that. We've been
avoiding that clear answer until now, creating tension between the
advocates of an ASF-like collection of tools and the advocates of a
tighter-integrated "openstack" product. We have created silos and
specialized areas as we got into the business of developing
time-series databases or SDNs. As a result, it's not "one community"
anymore. Should we further encourage that, or should we focus on
what the core of our mission is, what we have in common, this
integration engine that binds all those other open source projects
into one programmable infrastructure solution ?

--
Thierry Carrez (ttx)

I think we would do better as an ASF-like organization with a bias
towards programmable infrastructure. I do not think we should include
more time-series databases, RDBMSs, or even Message Brokers simply for
the sake of inclusion (there are examples of these in the big tent
already). But we should be careful about excluding and eliminating
projects and scope as well. We should include golang, and work on the
other issues separately (this sounds like something we should be working
on setting proper guideposts for the community, how we do that, etc, and
revolves around improving how the TC provides leadership). As part of
improving the leadership of OpenStack, we need to also work to have a
clear product-vision (and I do not mean "product" as in something
specifically sell-able). I think part of our issue and what is driving
these conversations is a lack of clear product vision which is part of
the TC providing the guideposts and improving leadership within OpenStack.

--
Morgan Fainberg (notmorgan)


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 19, 2016 by harlowja_at_fastmail (16,200 points)   2 5 8
0 votes

John Dickinson wrote:
summary:
* Defining the scope of OpenStack projects DOES NOT define the
languages needed to implement them. The considerations are
orthogonal.
* We've already defined OpenStack--it's whatever it takes to
fulfill its mission statement.

On 19 May 2016, at 6:19, Thierry Carrez wrote:

Hi everyone,

The discussion on the addition of golang focuses on estimating
community costs vs. technical benefits, so that the TC can make the
right call for "OpenStack". From that discussion, we established
that the costs for cross-project teams (Infra...) seem to be
workable. There is still significant community fragmentation
effects as we add another language, creating language-expert silos,
duplicating efforts, and losing some productivity as some
needlessly rewrite things in golang. But we seem generally ready to
accept that cost because we are missing a tool in our toolbelt: a
language that lets us do such native optimization.

We have a number of projects in OpenStack that are more low-level
than others and which require such optimization, and for those
projects (or subprojects) our current mix of language is not
satisfactory. The Swift team in particular has spent a lot of time
trying to get the I/O behavior they require with hard disk access
using Python, and they certainly did not jump on the golang
bandwagon lightly.

I believe the programming languages you need in OpenStack official
projects are a function of the scope you define for OpenStack
official projects. We wouldn't need to have JavaScript in the mix if
we considered that web interfaces that purely consume OpenStack
APIs are projects that consume OpenStack, rather than part of
OpenStack itself.

In the same vein, if we consider lower-level projects (which often
require such native optimization) as part of "OpenStack", rather
than as external open source projects that should be integrated by
"OpenStack", then we need a language like golang in our toolbelt.
There is basically no point in saying no to golang in OpenStack if
we need lower-level native optimization in OpenStack: we'll have to
accept the community cost that comes with such a community scope.

Defining "lower-level" is very hard. Since the Nova API[1] is
listening to a public network interface and coordinating with various
services in a cluster, is it low-level enough to need to consider
optimizations? Does the Nova API require optimization to handle a very
large number of connections using all of the hardware available on a
single server? If Nova is going to eek out every drop of performance
possible from a server, it probably does need to consider all kinds of
"low-level" optimizations.[2]

Because deployers of any OpenStack project do want efficient software
that is performant, all parts of OpenStack need to consider what it
takes to meet the performance demanded. Most of the time this will not
require rewriting code in a different language (that's almost never
the right answer), but my point is that I believe you're wrong that
defining the scope of OpenStack projects also defines the languages
needed to implement them. The considerations are orthogonal.

[1] substitute your favorite OpenStack project

[2] http://highscalability.com/blog/2013/5/13/the-secret-to-10-million-concurrent-connections-the-kernel-i.html

So the real question we need to answer is... where does OpenStack
stop, and where does the wider open source community start ? If
OpenStack is purely an "integration engine", glue code for other
lower-level technologies like hypervisors, databases, or distributed
block storage, then the scope is limited, Python should be plenty
enough, and we don't need to fragment our community. If OpenStack is
"whatever it takes to reach our mission", then yes we need to add one
language to cover lower-level/native optimization, because we'll
need that... and we need to accept the community cost as a
consequence of that scope choice. Those are the only two options on
the table.

I'm actually not sure what is the best answer. But I'm convinced we,
as a community, need to have a clear answer to that. We've been
avoiding that clear answer until now, creating tension between the
advocates of an ASF-like collection of tools and the advocates of a
tighter-integrated "openstack" product. We have created silos and
specialized areas as we got into the business of developing time-
series databases or SDNs. As a result, it's not "one community"
anymore. Should we further encourage that, or should we focus on
what the core of our mission is, what we have in common, this
integration engine that binds all those other open source projects
into one programmable infrastructure solution ?

You said the answer in your question. OpenStack isn't defined as an
integration engine[3]. The definition of OpenStack is whatever it
takes to fulfill our mission[4][5]. I don't mean that as a tautology.
I mean that we've already gone to the effort of defining OpenStack. It's
our mission statement. We're all about building a cloud platform upon
which people can run their apps ("cloud-native" or otherwise), so we
write the software needed to do that.

So where does OpenStack stop and the wider community start? OpenStack
includes the projects needed to fulfill its mission.

[3] There are foundation staff who have used that phrase to help
describe some of the work we do (mostly to explain how OpenStack still
matters in a docker/kubernetes/container world). While that may be good for
explaining and bringing people in, it's not the definition of
OpenStack.

And to expand this. IMHO its not even a good way of explaining 'how
OpenStack still matters in a docker/kubernetes/container world' because
that world doesn't fully exist yet (and likely won't for years) and
already saying that the way for OpenStack to work here is via an
integration engine is sad.

This is especially wrong because those docker/kubernetes are still
maturing projects and people (from both communities) are working on
connecting the two different worlds together in a way that makes sense
(and that work is not frozen in time)
[seehttps://groups.google.com/forum/#!forum/kubernetes-sig-openstack for
one such variant]. One take on that work progressing is that kubernetes
could be just another project that uses keystone, cinder, neutron ...
(in a similar vein to how nova uses those same projects).

[4] "To produce a ubiquitous Open Source Cloud Computing platform that
is easy to use, simple to implement, interoperable between
deployments, works well at all scales, and meets the needs of users
and operators of both public and private clouds."

[5] From openstack.org: "OpenStack software controls large pools of
compute, storage, and networking resources throughout a datacenter,
managed through a dashboard or via the OpenStack API."


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 19, 2016 by harlowja_at_fastmail (16,200 points)   2 5 8
0 votes

+1. OpenStack does meet the requirements for being an Operating System at the data center level in my opinion. We just keep trying to beat around that bush for some reason... yes, we do that by Integrating a lot of things together, but thats not what its about.


From: Joshua Harlow [harlowja@fastmail.com]
Sent: Thursday, May 19, 2016 3:01 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [all][tc] Languages vs. Scope of "OpenStack"

Morgan Fainberg wrote:

On Thu, May 19, 2016 at 6:19 AM, Thierry Carrez <thierry@openstack.org
thierry@openstack.org> wrote:

Hi everyone,

The discussion on the addition of golang focuses on estimating
community costs vs. technical benefits, so that the TC can make the
right call for "OpenStack". From that discussion, we established
that the costs for cross-project teams (Infra...) seem to be
workable. There is still significant community fragmentation effects
as we add another language, creating language-expert silos,
duplicating efforts, and losing some productivity as some needlessly
rewrite things in golang. But we seem generally ready to accept that
cost because we are missing a tool in our toolbelt: a language that
lets us do such native optimization.

We have a number of projects in OpenStack that are more low-level
than others and which require such optimization, and for those
projects (or subprojects) our current mix of language is not
satisfactory. The Swift team in particular has spent a lot of time
trying to get the I/O behavior they require with hard disk access
using Python, and they certainly did not jump on the golang
bandwagon lightly.

I agree with this. it is totally reasonable to add golang (or a tool to
fill the need for native optimizations). The fragmentation concerns are
real, but filling this gap in the openstack tool-chain is important. It
will garner more interest in the project as it opens the doors to
examine the native level optimizations that are exceptionally hard in
Python. While this all could be done in Python, I would argue that there
are legitimate cases where a tool like golang is going to be a "better"
fit for the job. I would also argue that the TC should provide a strong
guidance on where golang should be used (initially). Specifically that
it should be used in cases where Python can demonstrably be shown to be
the inferior tool for the job and not as whole-sale replacement(s) for
the core API/end-user interactions. The type of break that the swift
team is advocating for is a great starting place. With the
acknowledgement from the -infra team that this is reasonable and
workable, I am willing (with my TC hat on), agree to include golang
provided the recommendations are outlined for the initial use-cases. As
we see where things shake out with these initial use-cases I am sure we
will see expanded uses of golang that make sense. I do know that
policing such uses is difficult at best, and I don't think it is
something that should be done via technology but more of a social
contract and documented as the recommended for initial cases of golang
(or other similar tool-chain) inclusion.

I believe the programming languages you need in OpenStack official
projects are a function of the scope you define for OpenStack
official projects. We wouldn't need to have JavaScript in the mix if
we considered that web interfaces that purely consume OpenStack APIs
are projects that consume OpenStack, rather than part of OpenStack
itself.

In the same vein, if we consider lower-level projects (which often
require such native optimization) as part of "OpenStack", rather
than as external open source projects that should be integrated by
"OpenStack", then we need a language like golang in our toolbelt.
There is basically no point in saying no to golang in OpenStack if
we need lower-level native optimization in OpenStack: we'll have to
accept the community cost that comes with such a community scope.

So the real question we need to answer is... where does OpenStack
stop, and where does the wider open source community start ? If
OpenStack is purely an "integration engine", glue code for other
lower-level technologies like hypervisors, databases, or distributed
block storage, then the scope is limited, Python should be plenty
enough, and we don't need to fragment our community. If OpenStack is
"whatever it takes to reach our mission", then yes we need to add
one language to cover lower-level/native optimization, because we'll
need that... and we need to accept the community cost as a
consequence of that scope choice. Those are the only two options on
the table.

It would be disingenuous to say we are a pure integration engine. If
swift or a similar project is outside the scope of openstack, frankly, I
view Keystone as outside of the scope of openstack. Keystone (while in a
different position due to a lot of heavy requirements for auth in
openstack) is not a simple "integrate an underlying technology with
other things with some tracking of the resources". Keystone provides a
real added value, while perhaps not as low-level as swift, that extends
significantly beyond the scope of "glue" of it's individual components.
I do believe the lower level technologies have a place in "openstack's
big tent" but should be evaluated carefully before inclusion; I am not
advocating we include MySQL, PGSQL, or RabbitMQ as "part of the big
tent". I think object storage is an important thing within the cloud
ecosystem, and by it's nature swift belongs as part of the big tent.
The cost for golang, in my view after having a large number of
conversations, is worth it. This conversation will continue to come up
in different forms if we nix golang here, the low level optimizations
are relevant and the projects (initially swift) that are running up
against these issues are in-fact a part of OpenStack today.. I also want
to make sure we, as the TC and community, are not evaluating a specific
project here, but OpenStack as a whole. If we are evaluating one project
specifically and making a change based upon where it is in the
ecosystem, we should step way back and evaluate why this just now has
become part of the conversation and address that issue first

+1

I really dislike the words 'integration engine' because IMHO its not
honest, its not interesting and I don't feel its really innovative and
attractive to developers that are coming to work on a opensource project
(for say there first time). If I was a new developer out of college and
was going to be working on a 'integration engine' (and/or glue engine)
I'd be running for the hills. We really need to work on a better terms
here IMHO ;)

I'm actually not sure what is the best answer. But I'm convinced we,
as a community, need to have a clear answer to that. We've been
avoiding that clear answer until now, creating tension between the
advocates of an ASF-like collection of tools and the advocates of a
tighter-integrated "openstack" product. We have created silos and
specialized areas as we got into the business of developing
time-series databases or SDNs. As a result, it's not "one community"
anymore. Should we further encourage that, or should we focus on
what the core of our mission is, what we have in common, this
integration engine that binds all those other open source projects
into one programmable infrastructure solution ?

--
Thierry Carrez (ttx)

I think we would do better as an ASF-like organization with a bias
towards programmable infrastructure. I do not think we should include
more time-series databases, RDBMSs, or even Message Brokers simply for
the sake of inclusion (there are examples of these in the big tent
already). But we should be careful about excluding and eliminating
projects and scope as well. We should include golang, and work on the
other issues separately (this sounds like something we should be working
on setting proper guideposts for the community, how we do that, etc, and
revolves around improving how the TC provides leadership). As part of
improving the leadership of OpenStack, we need to also work to have a
clear product-vision (and I do not mean "product" as in something
specifically sell-able). I think part of our issue and what is driving
these conversations is a lack of clear product vision which is part of
the TC providing the guideposts and improving leadership within OpenStack.

--
Morgan Fainberg (notmorgan)


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 19, 2016 by Fox,_Kevin_M (29,360 points)   1 3 4
0 votes

John Dickinson wrote:

[...]
In the same vein, if we consider lower-level projects (which often
require such native optimization) as part of "OpenStack", rather
than as external open source projects that should be integrated by
"OpenStack", then we need a language like golang in our toolbelt.
There is basically no point in saying no to golang in OpenStack if
we need lower-level native optimization in OpenStack: we'll have to
accept the community cost that comes with such a community scope.

Defining "lower-level" is very hard. Since the Nova API[1] is
listening to a public network interface and coordinating with various
services in a cluster, is it low-level enough to need to consider
optimizations? Does the Nova API require optimization to handle a very
large number of connections using all of the hardware available on a
single server? If Nova is going to eek out every drop of performance
possible from a server, it probably does need to consider all kinds of
"low-level" optimizations.[2]

That is fair. There is no clear line between lower-level and
higher-level (although one might argue there is a line between projects
requiring Go and projects that don't require it). It's more of a gradient.

[...]

So the real question we need to answer is... where does OpenStack
stop, and where does the wider open source community start ? If
OpenStack is purely an "integration engine", glue code for other
lower-level technologies like hypervisors, databases, or distributed
block storage, then the scope is limited, Python should be plenty
enough, and we don't need to fragment our community. If OpenStack is
"whatever it takes to reach our mission", then yes we need to add one
language to cover lower-level/native optimization, because we'll
need that... and we need to accept the community cost as a
consequence of that scope choice. Those are the only two options on
the table.

I'm actually not sure what is the best answer. But I'm convinced we,
as a community, need to have a clear answer to that. We've been
avoiding that clear answer until now, creating tension between the
advocates of an ASF-like collection of tools and the advocates of a
tighter-integrated "openstack" product. We have created silos and
specialized areas as we got into the business of developing time-
series databases or SDNs. As a result, it's not "one community"
anymore. Should we further encourage that, or should we focus on
what the core of our mission is, what we have in common, this
integration engine that binds all those other open source projects
into one programmable infrastructure solution ?

You said the answer in your question. OpenStack isn't defined as an
integration engine[3]. The definition of OpenStack is whatever it
takes to fulfill our mission[4][5]. I don't mean that as a tautology.
I mean that we've already gone to the effort of defining OpenStack. It's
our mission statement. We're all about building a cloud platform upon
which people can run their apps ("cloud-native" or otherwise), so we
write the software needed to do that.

So where does OpenStack stop and the wider community start? OpenStack
includes the projects needed to fulfill its mission.

I'd totally agree with you if OpenStack was developed in a vacuum. But
there is a large number of open source projects and libraries that
OpenStack needs to fulfill its mission that are not in "OpenStack": they
are external open source projects we depend on. Python, MySQL, libvirt,
KVM, Ceph, OpenvSwitch, RabbitMQ... We are not asking that those should
be included in OpenStack, and we are not NIHing replacements for those
in OpenStack either.

So it is not as clear-cut as you present it, and you can approach this
dependency question from two directions.

One is community-centric: "anything produced by our community is
OpenStack". If we are missing a lower-level piece to achieve our mission
and are developing it ourselves as a result, then it is OpenStack, even
if it ends up being a message queue or a database.

The other approach is product-centric: "lower-level pieces are OpenStack
dependencies, rather than OpenStack itself". If we are missing a
lower-level piece to achieve our mission and are developing it as a
result, it could be developed on OpenStack infrastructure by members of
the OpenStack community but it is not "OpenStack the product", it's an
OpenStack dependency. It is not governed by the TC, it can use any
language and tool deemed necessary.

On this second approach, there is the obvious question of where
"lower-level" starts, which as you explained above is not really
clear-cut. A good litmus test for it could be whenever Python is not
enough. If you can't develop it effectively with the language that is
currently sufficient for the rest of OpenStack, then developing it as an
OpenStack dependency in whatever language is appropriate might be the
solution...

That is what I mean by 'scope': where does "OpenStack" stop, and where
do "OpenStack dependencies" start ? It is a lot easier and a lot less
community-costly to allow additional languages in OpenStack dependencies
(we already have plenty there).

--
Thierry Carrez (ttx)


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 20, 2016 by Thierry_Carrez (57,480 points)   3 8 13
0 votes

On Fri, 20 May 2016, Thierry Carrez wrote:

The other approach is product-centric: "lower-level pieces are OpenStack
dependencies, rather than OpenStack itself". If we are missing a lower-level
piece to achieve our mission and are developing it as a result, it could be
developed on OpenStack infrastructure by members of the OpenStack community
but it is not "OpenStack the product", it's an OpenStack dependency. It is
not governed by the TC, it can use any language and tool deemed necessary.

On this second approach, there is the obvious question of where "lower-level"
starts, which as you explained above is not really clear-cut. A good litmus
test for it could be whenever Python is not enough. If you can't develop it
effectively with the language that is currently sufficient for the rest of
OpenStack, then developing it as an OpenStack dependency in whatever language
is appropriate might be the solution...

I don't think language does (or should) have anything to do with it.

The question is whether or not the tool (whether service or
dependent library) is useful to and usable outside the openstack-stack.
For example gnocchi is useful to openstack but you can use it with other
stuff, therefore not openstack. More controversially: swift can be
usefully used all by its lonesome: not openstack.

Not being in OpenStack (where "in" means "of the product") is good
for OpenStack, good for the project and good for opensource in general:

  • Outside the OpenStack bubble, looking in, one can see a bunch of
    complexity and a bunch of bad architecture decisions but rarely
    see the good stuff that is actually there, so it is easy enough to walk
    away. Good stuff that a larger audience could benefit from may get
    dismissed, if that good stuff has an opportunity to have an
    independent identity, it can be useful.

  • A project that is used by a larger and more diverse audience
    (people-wise and technology-wise) will of necessity be more
    robust.

  • A project that defines itself as independent will be required to
    have strong and narrow contracts to satisfy its diverse audiences.

  • A project that has those strong and narrow contracts can use what
    ever language it likes and still be useful and nobody really needs
    to care all that deeply except for the people making it. If they
    want to be in a language that infra doesn't want to support,
    that's fine: there are plenty of other ways to do CI.

  • An openstack which is more narrow is far easier for people to
    comprehend and contemplate.

  • A broad opensource world that has lots of nice things is a better
    one.

--
Chris Dent (╯°□°)╯︵┻━┻ http://anticdent.org/
freenode: cdent tw: @anticdent__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

responded May 20, 2016 by cdent_plus_os_at_ant (12,800 points)   2 2 5
0 votes

On Fri, May 20, 2016 at 7:48 AM, Thierry Carrez thierry@openstack.org
wrote:

The other approach is product-centric: "lower-level pieces are OpenStack
dependencies, rather than OpenStack itself". If we are missing a
lower-level piece to achieve our mission and are developing it as a result,
it could be developed on OpenStack infrastructure by members of the
OpenStack community but it is not "OpenStack the product", it's an
OpenStack dependency. It is not governed by the TC, it can use any
language and tool deemed necessary.

I think we should include the degree of OpenStack-specificness here, as
something that may fit every other of your criteria for 'used by but not
part of OpenStack' is really only useful to OpenStack (Designate's DNS
code, for example, apparently reads the DB) should be part of OpenStack.
IIRC that has been used as one of the criteria for deciding if a new
library should be part of Oslo or stand-alone (independent of in-or-out of
OpenStack governance).

That is what I mean by 'scope': where does "OpenStack" stop, and where do
"OpenStack dependencies" start ? It is a lot easier and a lot less
community-costly to allow additional languages in OpenStack dependencies
(we already have plenty there).

So down the road when one of these dependencies that is very important to
OpenStack goes more dormant than we would like due to resource allocation
issues because it is not-Big Tent, will we adopt it like we have done with
other dependencies where that made more sense than re-writing around it?

I do hope that should the TC adopt the position of drawing the scope line
tighter around the core, that the tent-cleaning will follow in both
directions, down toward kernel-space and up toward end-user-space. We are
historically bad at leaving that sort of debt lying and cleaning can make
some strides toward reducing the community cost of maintaining the current
ecosystem.

dt

--

Dean Troyer
dtroyer@gmail.com


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 20, 2016 by Dean_Troyer (13,100 points)   1 3 3
0 votes

On Fri, May 20, 2016 at 8:16 AM, Chris Dent cdent+os@anticdent.org wrote:

I don't think language does (or should) have anything to do with it.

++++^1024

Language is what finally forced this discussion (as a prerequisite), now
that we're here, lets finish the prerequisite before going back to what got
us here.

The question is whether or not the tool (whether service or
dependent library) is useful to and usable outside the openstack-stack.
For example gnocchi is useful to openstack but you can use it with other
stuff, therefore not openstack. More controversially: swift can be
usefully used all by its lonesome: not openstack.

Heh, projects usually see this as a positive.

Not being in OpenStack (where "in" means "of the product") is good
for OpenStack, good for the project and good for opensource in general:

[excellent list of project attributes removed]

I would really hope these could also apply to OpenStack projects, and
understand why they usually don't. In or out of the tent, those attributes
have at least one strong common thread, the ability to say no when
necessary and follow a specific path. Self-discipline. In or out of the
tent, projects without that disintegrate eventually. I'm posting that list
on the clubhouse wall.

Since we are, by definition in the mission statement, all things to all
people, we got really inclusive in the last year or two. The pendulum does
appear to be swinging the other way however, somewhat due to the costs of
scale. Maybe entry into the tent should come with a ticket price to help
defray some of those costs? Not just direct $$$ (corporate foundation
membership), but dedicated X hours per cycle for documentation or Y hours
for Infra or Z core/time units of cloud resources. Wait, we do something
like that already? Maybe its time for a cost-of-living adjustment...

dt

--

Dean Troyer
dtroyer@gmail.com


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 20, 2016 by Dean_Troyer (13,100 points)   1 3 3
0 votes

Excerpts from Chris Dent's message of 2016-05-20 14:16:15 +0100:

On Fri, 20 May 2016, Thierry Carrez wrote:

The other approach is product-centric: "lower-level pieces are OpenStack
dependencies, rather than OpenStack itself". If we are missing a lower-level
piece to achieve our mission and are developing it as a result, it could be
developed on OpenStack infrastructure by members of the OpenStack community
but it is not "OpenStack the product", it's an OpenStack dependency. It is
not governed by the TC, it can use any language and tool deemed necessary.

On this second approach, there is the obvious question of where "lower-level"
starts, which as you explained above is not really clear-cut. A good litmus
test for it could be whenever Python is not enough. If you can't develop it
effectively with the language that is currently sufficient for the rest of
OpenStack, then developing it as an OpenStack dependency in whatever language
is appropriate might be the solution...

I don't think language does (or should) have anything to do with it.

The question is whether or not the tool (whether service or
dependent library) is useful to and usable outside the openstack-stack.
For example gnocchi is useful to openstack but you can use it with other
stuff, therefore not openstack. More controversially: swift can be
usefully used all by its lonesome: not openstack.

Add keystone, cinder, and ironic to that list.

Not being in OpenStack (where "in" means "of the product") is good
for OpenStack, good for the project and good for opensource in general:

  • Outside the OpenStack bubble, looking in, one can see a bunch of
    complexity and a bunch of bad architecture decisions but rarely
    see the good stuff that is actually there, so it is easy enough to walk
    away. Good stuff that a larger audience could benefit from may get
    dismissed, if that good stuff has an opportunity to have an
    independent identity, it can be useful.

  • A project that is used by a larger and more diverse audience
    (people-wise and technology-wise) will of necessity be more
    robust.

  • A project that defines itself as independent will be required to
    have strong and narrow contracts to satisfy its diverse audiences.

  • A project that has those strong and narrow contracts can use what
    ever language it likes and still be useful and nobody really needs
    to care all that deeply except for the people making it. If they
    want to be in a language that infra doesn't want to support,
    that's fine: there are plenty of other ways to do CI.

  • An openstack which is more narrow is far easier for people to
    comprehend and contemplate.

  • A broad opensource world that has lots of nice things is a better
    one.

As you say, there are a lot of good reasons to strive for loose
coupling between components. On the other hand, whether or not an
individual component can or should be used by itself isn't a
sufficient line when we're talking about the nature of OpenStack
as a whole, especially if we want interoperability between deployments.

Most of our services add value on top of some other existing project
(either by integrating them or abstracting them or both). What
would that story look like if we said that because keystone can be
used by itself, it is not part of OpenStack? Would it make sense
to still require deployments that want to use the trademark to use
keystone for authentication?

Doug


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded May 23, 2016 by Doug_Hellmann (87,520 points)   3 4 9
...