settingsLogin | Registersettings

[Openstack-operators] Packaging sample config versions

0 votes

Hello all,

Since somewhere around icehouse projects have started to stop shipping sample configuration files with their projects and instead create a README-sample.conf.txt that usually contains something like:
To generate the sample nova.conf file, run the following
command from the top level of the nova directory:

tox -egenconfig

The problem that I am running into is that tox -egenconfig now requires a newer version of tox than what is available for the build distro:
[root at localhost ceilometer-2014.2.1]# tox -egenconfig
ERROR: tox version is 1.4.2, required is at least 1.6

I know I can do a pip install blah and blindly hope that I get everything installed locally on my build system and that I don't install something that conflicts with the system... but that seems like a less than desirable solution. So how are people who are doing packaging handling this? Are you now building a venv per package for tox just to generate a sample configuration? Shouldn't this be part of the python build/install steps per project?

It seems redhat/rdo is simply including a sample configuration that they generated somehow.

What are you other packagers doing?

Also, is it just me or does this seem wrong? Most of the commits that made this change seem to indicate it was because the sample config file was separate from the project and that it was breaking gate when it wasn't kept up to date. Shouldn't this be something that each project generates? This seemed to work for 8 previous releases - now its too difficult? I don't get the motivations behind this change.


Kris Lindgren
Senior Linux Systems Engineer
GoDaddy, LLC.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.openstack.org/pipermail/openstack-operators/attachments/20141209/c9b8b829/attachment.html

asked Dec 9, 2014 in openstack-operators by Kris_G._Lindgren (7,740 points)   1 5 10

37 Responses

0 votes

I also find this issue really really annoying. Not only is the sample config a useful reference, but having it in a git repo allows me to see when new options were added or defaults were changed. Otherwise you have to rely on the docs which are generally wrong. When I was looking at some of the Juno transitions, I?ve already filed a few doc bugs about wrong defaults and config options. Assuming a project enforces that the sample config is up-to-date when commits are made, it?s by far the best source of this info. As for why? Well I?ve brought this up a few times before, and there was even a bug filed (https://bugs.launchpad.net/nova/+bug/1301519) but it was marked as ?Won?t Fix?. I don?t really know the reason. This is one of my larger annoyances as an operator?

I do know that some projects, like nova and cinder, still ship the file.

Maybe someone can setup a cron job to do a pull, build the sample configs and post them to git hub every day.

From: "Kris G. Lindgren" >
Date: Monday, December 8, 2014 at 7:46 PM
To: "openstack-operators at lists.openstack.org" >
Subject: [Openstack-operators] Packaging sample config versions

Hello all,

Since somewhere around icehouse projects have started to stop shipping sample configuration files with their projects and instead create a README-sample.conf.txt that usually contains something like:
To generate the sample nova.conf file, run the following
command from the top level of the nova directory:

tox ?egenconfig

The problem that I am running into is that tox ?egenconfig now requires a newer version of tox than what is available for the build distro:
[root at localhost ceilometer-2014.2.1]# tox -egenconfig
ERROR: tox version is 1.4.2, required is at least 1.6

I know I can do a pip install blah and blindly hope that I get everything installed locally on my build system and that I don?t install something that conflicts with the system? but that seems like a less than desirable solution. So how are people who are doing packaging handling this? Are you now building a venv per package for tox just to generate a sample configuration? Shouldn't this be part of the python build/install steps per project?

It seems redhat/rdo is simply including a sample configuration that they generated somehow.

What are you other packagers doing?

Also, is it just me or does this seem wrong? Most of the commits that made this change seem to indicate it was because the sample config file was separate from the project and that it was breaking gate when it wasn't kept up to date. Shouldn't this be something that each project generates? This seemed to work for 8 previous releases ? now its too difficult? I don?t get the motivations behind this change.


Kris Lindgren
Senior Linux Systems Engineer
GoDaddy, LLC.


This E-mail and any of its attachments may contain Time Warner Cable proprietary information, which is privileged, confidential, or subject to copyright belonging to Time Warner Cable. This E-mail is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient of this E-mail, you are hereby notified that any dissemination, distribution, copying, or action taken in relation to the contents of and attachments to this E-mail is strictly prohibited and may be unlawful. If you have received this E-mail in error, please notify the sender immediately and permanently delete the original and any copy of this E-mail and any printout.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
responded Dec 9, 2014 by Fischer,_Matt (1,380 points)   1 3
0 votes

Fischer, Matt wrote:
I also find this issue really really annoying. Not only is the sample
config a useful reference, but having it in a git repo allows me to see
when new options were added or defaults were changed. Otherwise you have
to rely on the docs which are generally wrong. When I was looking at
some of the Juno transitions, I?ve already filed a few doc bugs about
wrong defaults and config options. Assuming a project enforces that the
sample config is up-to-date when commits are made, it?s by far the best
source of this info. As for why? Well I?ve brought this up a few times
before, and there was even a bug filed
(https://bugs.launchpad.net/nova/+bug/1301519
https://bugs.launchpad.net/nova/+bug/1301519) but it was marked as
?Won?t Fix?. I don?t really know the reason. This is one of my larger
annoyances as an operator?

I do know that some projects, like nova and cinder, still ship the file.

Cinder I believe just kicked that out (for better or worse),

https://github.com/openstack/cinder/commit/62a72a7574b18f7

One less. I don't think nova does either anymore...

Personally I don't know how to solve this without changes around the
usage of oslo.config and its pervasiveness (and how usage of oslo.config
by libraries can alter the valid configuration of a project just by
installing a different library version); which is a pretty big change
all over the place.

Perhaps this goes back to the point that I've seen on the developers
list in:
http://lists.openstack.org/pipermail/openstack-dev/2014-December/052150.html
(perhaps projects shouldn't use libraries that use oslo.config and those
projects should only use oslo.config in there top-level project and
configure libraries via some other mechanism that doesn't cause these
kinds of dynamically changing configuration issues).

Something to think about...

-Josh

Maybe someone can setup a cron job to do a pull, build the sample
configs and post them to git hub every day.

From: "Kris G. Lindgren" >
Date: Monday, December 8, 2014 at 7:46 PM
To: "openstack-operators at lists.openstack.org
"
>
Subject: [Openstack-operators] Packaging sample config versions

Hello all,

Since somewhere around icehouse projects have started to stop shipping
sample configuration files with their projects and instead create a
README-sample.conf.txt that usually contains something like:
To generate the sample nova.conf file, run the following
command from the top level of the nova directory:

tox ?egenconfig

The problem that I am running into is that tox ?egenconfig now requires
a newer version of tox than what is available for the build distro:
[root at localhost ceilometer-2014.2.1]# tox -egenconfig
ERROR: tox version is 1.4.2, required is at least 1.6

I know I can do a pip install blah and blindly hope that I get
everything installed locally on my build system and that I don?t install
something that conflicts with the system? but that seems like a less
than desirable solution. So how are people who are doing packaging
handling this? Are you now building a venv per package for tox just to
generate a sample configuration? Shouldn't this be part of the python
build/install steps per project?

It seems redhat/rdo is simply including a sample configuration that they
generated somehow.

What are you other packagers doing?

Also, is it just me or does this seem wrong? Most of the commits that
made this change seem to indicate it was because the sample config file
was separate from the project and that it was breaking gate when it
wasn't kept up to date. Shouldn't this be something that each project
generates? This seemed to work for 8 previous releases ? now its too
difficult? I don?t get the motivations behind this change.


Kris Lindgren
Senior Linux Systems Engineer
GoDaddy, LLC.


This E-mail and any of its attachments may contain Time Warner Cable
proprietary information, which is privileged, confidential, or subject
to copyright belonging to Time Warner Cable. This E-mail is intended
solely for the use of the individual or entity to which it is addressed.
If you are not the intended recipient of this E-mail, you are hereby
notified that any dissemination, distribution, copying, or action taken
in relation to the contents of and attachments to this E-mail is
strictly prohibited and may be unlawful. If you have received this
E-mail in error, please notify the sender immediately and permanently
delete the original and any copy of this E-mail and any printout.


OpenStack-operators mailing list
OpenStack-operators at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
responded Dec 9, 2014 by Joshua_Harlow (12,560 points)   1 4 4
0 votes

On 2014-12-08 10:23 PM, Fischer, Matt wrote:

I do know that some projects, like nova and cinder, still ship the file.

Nova does not ship the config file anymore:
https://review.openstack.org/#/c/81588/
https://github.com/openstack/nova/blob/master/etc/nova/README-nova.conf.txt

Cinder recently removed its file too:
https://review.openstack.org/#/c/139511/
https://github.com/openstack/cinder/commit/62a72a7574b18f74e8fceab700c2e831f6ca8961

--
Mathieu

responded Dec 9, 2014 by Mathieu_Gagné (3,300 points)   1 3 6
0 votes

I do know that some projects, like nova and cinder, still ship the file.

Cinder I believe just kicked that out (for better or worse),

https://github.com/openstack/cinder/commit/62a72a7574b18f7

One less. I don't think nova does either anymore...

Oops I meant keystone! Nova was one of the first places I noticed it was
gone.

This E-mail and any of its attachments may contain Time Warner Cable proprietary information, which is privileged, confidential, or subject to copyright belonging to Time Warner Cable. This E-mail is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient of this E-mail, you are hereby notified that any dissemination, distribution, copying, or action taken in relation to the contents of and attachments to this E-mail is strictly prohibited and may be unlawful. If you have received this E-mail in error, please notify the sender immediately and permanently delete the original and any copy of this E-mail and any printout.

responded Dec 9, 2014 by Fischer,_Matt (1,380 points)   1 3
0 votes

Matt,

I completely agree.

Nova stopped shipping the file starting in icehouse, ceilometer started in Juno, I see commits now in pretty much every project that remove the sample config and replace it with some tox script. Which may or may not run on your current operating system ? without going to pip to get newer versions of stuff.

I see that whats being said is that a valid sample configuration file depends on the version of the other dependent libraries that are installed on the system as well. I assume this is talking about different versions of oslo.config and oslo.messaging take different configuration options and as such we can't generate a generic sample config file because we don?t know what versions of dependent libraries you actually have installed.

This seems to have taken a problem that?s been created on the development side and simply punted over the wall to let document writers/operators/packagers deal with: As this sample configuration file changes independently from the project, this check failed on the gate time to time, as this file needed to be updated manually [1]. I guess I reject the assertion that the sample configuration file is independent of the project. Just because some options change depending on the oslo.config version doesn't mean the configuration options that are specific to that project change as well?

Anyway ? back to packaging. The problem that I have with this change is that in order to package a sample configuration I need to basically do the following: 1.) get the current build/commit run 2.) run python build 3.) strip away the relevant built parts of the package 4.) install on the build machine all the python runtime deps that I am going to need for this package 5.) install tox and other packages as needed to generate a sample configuration 6.) generate sample config 7.) build a new package with the exact same requirements as what I installed in step 4 8.) add sample configuration generated in step 6 to the package. Then I need to make sure I also package all of the python-versions that was used in step 4, making sure that I don?t have conflicting system level dependencies from other openstack projects.

Now anvil can do a lot of this for me? but this seems overly complicated.

I don?t think its too much to ask for each project to include a script that will build a venv that includes tox and the other relevant deps to build the sample configuration. IMHO, its also ok if the sample configuration is not 100% perfect re: config options for dependencies ? call that out in the config file and we can look up what the correct config parameters are based upon our configuration environment. But there is no execuse to not include an example config that at least has the config options relevant to project.


Kris Lindgren
Senior Linux Systems Engineer
GoDaddy, LLC.

From: , Matt <matthew.fischer at twcable.com<mailto:matthew.fischer at twcable.com>>
Date: Monday, December 8, 2014 at 8:23 PM
To: "Kris G. Lindgren" >, "openstack-operators at lists.openstack.org" >
Subject: Re: [Openstack-operators] Packaging sample config versions

I also find this issue really really annoying. Not only is the sample config a useful reference, but having it in a git repo allows me to see when new options were added or defaults were changed. Otherwise you have to rely on the docs which are generally wrong. When I was looking at some of the Juno transitions, I?ve already filed a few doc bugs about wrong defaults and config options. Assuming a project enforces that the sample config is up-to-date when commits are made, it?s by far the best source of this info. As for why? Well I?ve brought this up a few times before, and there was even a bug filed (https://bugs.launchpad.net/nova/+bug/1301519) but it was marked as ?Won?t Fix?. I don?t really know the reason. This is one of my larger annoyances as an operator?

I do know that some projects, like nova and cinder, still ship the file.

Maybe someone can setup a cron job to do a pull, build the sample configs and post them to git hub every day.

From: "Kris G. Lindgren" >
Date: Monday, December 8, 2014 at 7:46 PM
To: "openstack-operators at lists.openstack.org" >
Subject: [Openstack-operators] Packaging sample config versions

Hello all,

Since somewhere around icehouse projects have started to stop shipping sample configuration files with their projects and instead create a README-sample.conf.txt that usually contains something like:
To generate the sample nova.conf file, run the following
command from the top level of the nova directory:

tox ?egenconfig

The problem that I am running into is that tox ?egenconfig now requires a newer version of tox than what is available for the build distro:
[root at localhost ceilometer-2014.2.1]# tox -egenconfig
ERROR: tox version is 1.4.2, required is at least 1.6

I know I can do a pip install blah and blindly hope that I get everything installed locally on my build system and that I don?t install something that conflicts with the system? but that seems like a less than desirable solution. So how are people who are doing packaging handling this? Are you now building a venv per package for tox just to generate a sample configuration? Shouldn't this be part of the python build/install steps per project?

It seems redhat/rdo is simply including a sample configuration that they generated somehow.

What are you other packagers doing?

Also, is it just me or does this seem wrong? Most of the commits that made this change seem to indicate it was because the sample config file was separate from the project and that it was breaking gate when it wasn't kept up to date. Shouldn't this be something that each project generates? This seemed to work for 8 previous releases ? now its too difficult? I don?t get the motivations behind this change.


Kris Lindgren
Senior Linux Systems Engineer
GoDaddy, LLC.


This E-mail and any of its attachments may contain Time Warner Cable proprietary information, which is privileged, confidential, or subject to copyright belonging to Time Warner Cable. This E-mail is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient of this E-mail, you are hereby notified that any dissemination, distribution, copying, or action taken in relation to the contents of and attachments to this E-mail is strictly prohibited and may be unlawful. If you have received this E-mail in error, please notify the sender immediately and permanently delete the original and any copy of this E-mail and any printout.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
responded Dec 9, 2014 by Kris_G._Lindgren (7,740 points)   1 5 10
0 votes

On 2014-12-08 11:01 PM, Kris G. Lindgren wrote:

I don?t think its too much to ask for each project to include a script
that will build a venv that includes tox and the other relevant deps to
build the sample configuration.

This is already the case. Back then, I did the work of documenting how
you could generate the sample config files for each projects (I cared
about):
http://blog.mgagne.ca/generating-sample-config-files-in-openstack/

You can see that the process isn't streamlined. Each project has its own
particularities. Some projects don't use the (un)official standard "tox
-egenconfig" command, I patched some projects to make it less a pain.

And my thoughts about sample config files:
http://blog.mgagne.ca/where-are-the-sample-config-files/

I wrote it after Cinder proposed removing its sample config file. They
abandoned the patch at that time but now sample config file is gone.

It's not an easy problem because core libraries used by OpenStack
projects also use oslo.config and configs required by those libraries
are part of the main configurations required for a project to even work.
([database]/connection for instance) You just can't ignore those configs
when generating the sample config file.

(sorry for self-promotion, I didn't want to rewrite my thoughts on this
list)

--
Mathieu

responded Dec 9, 2014 by Mathieu_Gagné (3,300 points)   1 3 6
0 votes

So more to my point on the latest version of RHEL and doing: yum install
tox -egenconfig

ceilometer-2014.2.1]# tox -egenconfig
ERROR: tox version is 1.4.2, required is at least 1.6

nova-2014.2.1]# tox -egenconfig
ERROR: tox version is 1.4.2, required is at least 1.6

glance-2014.2.1]# tox -egenconfig
ERROR: tox version is 1.4.2, required is at least 1.6

[root at localhost ~]# pip install --update tox
(Updated tox to 1.8.1 , upgraded virtualenv to 1.10.1 and upgraded py to
1.4.14)

glance-2014.2.1]# tox -egenconfig
genconfig create: /root/rpmbuild/BUILD/glance-2014.2.1/.tox/genconfig
genconfig installdeps:
-r/root/rpmbuild/BUILD/glance-2014.2.1/requirements.txt,
-r/root/rpmbuild/BUILD/glance-2014.2.1/test-requirements.txt
ERROR: invocation failed (exit code 1), logfile:
/root/rpmbuild/BUILD/glance-2014.2.1/.tox/genconfig/log/genconfig-1.log
ERROR: actionid=genconfig

Running setup.py install for MySQL-python

/usr/include/mysql/myconfigx8664.h:654:2: error: #error
<my
config.h> MUST be included first!
#error <my_config.h> MUST be included first!
^
error: command 'gcc' failed with exit status 1

__________________________________________________________________ summary


ERROR: genconfig: could not install deps
[-r/root/rpmbuild/BUILD/glance-2014.2.1/requirements.txt,
-r/root/rpmbuild/BUILD/glance-2014.2.1/test-requirements.txt]; v =
InvocationError('/root/rpmbuild/BUILD/glance-2014.2.1/.tox/genconfig/bin/pi
p install --allow-all-external --allow-insecure netaddr -U
-r/root/rpmbuild/BUILD/glance-2014.2.1/requirements.txt
-r/root/rpmbuild/BUILD/glance-2014.2.1/test-requirements.txt (see
/root/rpmbuild/BUILD/glance-2014.2.1/.tox/genconfig/log/genconfig-1.log)',
1)

So a few things to point out in order to even get tox -egenconfig I had to
update the system packages versions using pip. Since we have other python
packages using virtualenv I have no idea if the updated venvironment
package is going to break those systems or not. So the included
script/command is already a barrier to getting a sample config. 2) tox
fails to even build all the deps - it happens to be exactly failing at
mysql in both nova/cinder/glance/keystone 3) It's installing it own
versions of python libraries that solve the dependencies that are then
going to be used to generate the configuration. If the configuration is
so dynamic that getting a different version of oslo.config could generate
a sample configuration that wont work on my system then how am I suppose
to deal with:
Tox installed version:
oslo.config-1.5.0

System installed version:
python-oslo-config-1.3.0

Also python-libvrit failed to build because I don?t have libvrit installed
on this system. So am I to assume that there are no libvrit options
(which we both know is false)?
Now I can get a example config - that wont work with my system - per what
everyone else has been saying. Also, at what point would the average user
just say "F it"? - because at the point I feel like if I was an average
user - I would be there right now.


Kris Lindgren
Senior Linux Systems Engineer
GoDaddy, LLC.

On 12/9/14, 8:14 AM, "Mathieu Gagn?" wrote:

On 2014-12-08 11:01 PM, Kris G. Lindgren wrote:

I don?t think its too much to ask for each project to include a script
that will build a venv that includes tox and the other relevant deps to
build the sample configuration.

This is already the case. Back then, I did the work of documenting how
you could generate the sample config files for each projects (I cared
about):
http://blog.mgagne.ca/generating-sample-config-files-in-openstack/

You can see that the process isn't streamlined. Each project has its own
particularities. Some projects don't use the (un)official standard "tox
-egenconfig" command, I patched some projects to make it less a pain.

And my thoughts about sample config files:
http://blog.mgagne.ca/where-are-the-sample-config-files/

I wrote it after Cinder proposed removing its sample config file. They
abandoned the patch at that time but now sample config file is gone.

It's not an easy problem because core libraries used by OpenStack
projects also use oslo.config and configs required by those libraries
are part of the main configurations required for a project to even work.
([database]/connection for instance) You just can't ignore those configs
when generating the sample config file.

(sorry for self-promotion, I didn't want to rewrite my thoughts on this
list)

--
Mathieu

responded Dec 9, 2014 by Kris_G._Lindgren (7,740 points)   1 5 10
0 votes

Well I think we can all agree this is an irritation. But how are others
actually dealing with this problem? (Maybe it?s less complicated in
Ubuntu.)

The sense I get is that most people using Anvil, or other custom-ish
packaging tools, are also running config management which handles
generating the config files, anyway. So you don?t so much care about the
contents of the config file shipped with the package.

Is that accurate for most people? Or are folks doing some other magic to
get a good config file in the packages?

Mike

On 12/9/14, 5:02 PM, "Kris G. Lindgren" wrote:

So more to my point on the latest version of RHEL and doing: yum install
tox -egenconfig

ceilometer-2014.2.1]# tox -egenconfig
ERROR: tox version is 1.4.2, required is at least 1.6

nova-2014.2.1]# tox -egenconfig
ERROR: tox version is 1.4.2, required is at least 1.6

glance-2014.2.1]# tox -egenconfig
ERROR: tox version is 1.4.2, required is at least 1.6

[root at localhost ~]# pip install --update tox
(Updated tox to 1.8.1 , upgraded virtualenv to 1.10.1 and upgraded py to
1.4.14)

glance-2014.2.1]# tox -egenconfig
genconfig create: /root/rpmbuild/BUILD/glance-2014.2.1/.tox/genconfig
genconfig installdeps:
-r/root/rpmbuild/BUILD/glance-2014.2.1/requirements.txt,
-r/root/rpmbuild/BUILD/glance-2014.2.1/test-requirements.txt
ERROR: invocation failed (exit code 1), logfile:
/root/rpmbuild/BUILD/glance-2014.2.1/.tox/genconfig/log/genconfig-1.log
ERROR: actionid=genconfig

Running setup.py install for MySQL-python

/usr/include/mysql/myconfigx8664.h:654:2: error: #error
<my
config.h> MUST be included first!
#error <my_config.h> MUST be included first!
^
error: command 'gcc' failed with exit status 1

__________________________________________________________________ summary
ERROR: genconfig: could not install deps
[-r/root/rpmbuild/BUILD/glance-2014.2.1/requirements.txt,
-r/root/rpmbuild/BUILD/glance-2014.2.1/test-requirements.txt]; v =
InvocationError('/root/rpmbuild/BUILD/glance-2014.2.1/.tox/genconfig/bin/p
i
p install --allow-all-external --allow-insecure netaddr -U
-r/root/rpmbuild/BUILD/glance-2014.2.1/requirements.txt
-r/root/rpmbuild/BUILD/glance-2014.2.1/test-requirements.txt (see
/root/rpmbuild/BUILD/glance-2014.2.1/.tox/genconfig/log/genconfig-1.log)',
1)

So a few things to point out in order to even get tox -egenconfig I had to
update the system packages versions using pip. Since we have other python
packages using virtualenv I have no idea if the updated venvironment
package is going to break those systems or not. So the included
script/command is already a barrier to getting a sample config. 2) tox
fails to even build all the deps - it happens to be exactly failing at
mysql in both nova/cinder/glance/keystone 3) It's installing it own
versions of python libraries that solve the dependencies that are then
going to be used to generate the configuration. If the configuration is
so dynamic that getting a different version of oslo.config could generate
a sample configuration that wont work on my system then how am I suppose
to deal with:
Tox installed version:
oslo.config-1.5.0

System installed version:
python-oslo-config-1.3.0

Also python-libvrit failed to build because I don?t have libvrit installed
on this system. So am I to assume that there are no libvrit options
(which we both know is false)?
Now I can get a example config - that wont work with my system - per what
everyone else has been saying. Also, at what point would the average user
just say "F it"? - because at the point I feel like if I was an average
user - I would be there right now.


Kris Lindgren
Senior Linux Systems Engineer
GoDaddy, LLC.

On 12/9/14, 8:14 AM, "Mathieu Gagn?" wrote:

On 2014-12-08 11:01 PM, Kris G. Lindgren wrote:

I don?t think its too much to ask for each project to include a
script
that will build a venv that includes tox and the other relevant deps to
build the sample configuration.

This is already the case. Back then, I did the work of documenting how
you could generate the sample config files for each projects (I cared
about):
http://blog.mgagne.ca/generating-sample-config-files-in-openstack/

You can see that the process isn't streamlined. Each project has its own
particularities. Some projects don't use the (un)official standard "tox
-egenconfig" command, I patched some projects to make it less a pain.

And my thoughts about sample config files:
http://blog.mgagne.ca/where-are-the-sample-config-files/

I wrote it after Cinder proposed removing its sample config file. They
abandoned the patch at that time but now sample config file is gone.

It's not an easy problem because core libraries used by OpenStack
projects also use oslo.config and configs required by those libraries
are part of the main configurations required for a project to even work.
([database]/connection for instance) You just can't ignore those configs
when generating the sample config file.

(sorry for self-promotion, I didn't want to rewrite my thoughts on this
list)

--
Mathieu


OpenStack-operators mailing list
OpenStack-operators at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
responded Dec 9, 2014 by Michael_Dorman (4,160 points)   5 11
0 votes

On Tue, Dec 9, 2014 at 11:25 AM, Michael Dorman wrote:

Well I think we can all agree this is an irritation. But how are others
actually dealing with this problem? (Maybe it?s less complicated in
Ubuntu.)

The sense I get is that most people using Anvil, or other custom-ish
packaging tools, are also running config management which handles
generating the config files, anyway. So you don?t so much care about the
contents of the config file shipped with the package.

Is that accurate for most people? Or are folks doing some other magic to
get a good config file in the packages?

The docs team -- reallly, Matt Kassawara -- regularly logs bugs for
packagers to put in better, working, default config files.

We do generate documentation for all the configurations across projects
that use oslo.config (and even for swift, which doesn't). So you can rely
on this reference:
http://docs.openstack.org/juno/config-reference/content/

You can also see new, updated, and deprecated options for each service,
such as:
http://docs.openstack.org/juno/config-reference/content/nova-conf-changes-juno.html

I don't believe our reference document was what encouraged devs to take the
sample config generation out-of-tree, but I am letting you know your best
option besides troubleshooting generating it yourself.

Anne

Mike

On 12/9/14, 5:02 PM, "Kris G. Lindgren" wrote:

So more to my point on the latest version of RHEL and doing: yum install
tox -egenconfig

ceilometer-2014.2.1]# tox -egenconfig
ERROR: tox version is 1.4.2, required is at least 1.6

nova-2014.2.1]# tox -egenconfig
ERROR: tox version is 1.4.2, required is at least 1.6

glance-2014.2.1]# tox -egenconfig
ERROR: tox version is 1.4.2, required is at least 1.6

[root at localhost ~]# pip install --update tox
(Updated tox to 1.8.1 , upgraded virtualenv to 1.10.1 and upgraded py to
1.4.14)

glance-2014.2.1]# tox -egenconfig
genconfig create: /root/rpmbuild/BUILD/glance-2014.2.1/.tox/genconfig
genconfig installdeps:
-r/root/rpmbuild/BUILD/glance-2014.2.1/requirements.txt,
-r/root/rpmbuild/BUILD/glance-2014.2.1/test-requirements.txt
ERROR: invocation failed (exit code 1), logfile:
/root/rpmbuild/BUILD/glance-2014.2.1/.tox/genconfig/log/genconfig-1.log
ERROR: actionid=genconfig

Running setup.py install for MySQL-python

/usr/include/mysql/myconfigx8664.h:654:2: error: #error
<my
config.h> MUST be included first!
#error <my_config.h> MUST be included first!
^
error: command 'gcc' failed with exit status 1

__________________________________________________________________ summary
ERROR: genconfig: could not install deps
[-r/root/rpmbuild/BUILD/glance-2014.2.1/requirements.txt,
-r/root/rpmbuild/BUILD/glance-2014.2.1/test-requirements.txt]; v =
InvocationError('/root/rpmbuild/BUILD/glance-2014.2.1/.tox/genconfig/bin/p
i
p install --allow-all-external --allow-insecure netaddr -U
-r/root/rpmbuild/BUILD/glance-2014.2.1/requirements.txt
-r/root/rpmbuild/BUILD/glance-2014.2.1/test-requirements.txt (see
/root/rpmbuild/BUILD/glance-2014.2.1/.tox/genconfig/log/genconfig-1.log)',
1)

So a few things to point out in order to even get tox -egenconfig I had to
update the system packages versions using pip. Since we have other python
packages using virtualenv I have no idea if the updated venvironment
package is going to break those systems or not. So the included
script/command is already a barrier to getting a sample config. 2) tox
fails to even build all the deps - it happens to be exactly failing at
mysql in both nova/cinder/glance/keystone 3) It's installing it own
versions of python libraries that solve the dependencies that are then
going to be used to generate the configuration. If the configuration is
so dynamic that getting a different version of oslo.config could generate
a sample configuration that wont work on my system then how am I suppose
to deal with:
Tox installed version:
oslo.config-1.5.0

System installed version:
python-oslo-config-1.3.0

Also python-libvrit failed to build because I don?t have libvrit installed
on this system. So am I to assume that there are no libvrit options
(which we both know is false)?
Now I can get a example config - that wont work with my system - per what
everyone else has been saying. Also, at what point would the average user
just say "F it"? - because at the point I feel like if I was an average
user - I would be there right now.


Kris Lindgren
Senior Linux Systems Engineer
GoDaddy, LLC.

On 12/9/14, 8:14 AM, "Mathieu Gagn?" wrote:

On 2014-12-08 11:01 PM, Kris G. Lindgren wrote:

I don?t think its too much to ask for each project to include a
script
that will build a venv that includes tox and the other relevant deps to
build the sample configuration.

This is already the case. Back then, I did the work of documenting how
you could generate the sample config files for each projects (I cared
about):
http://blog.mgagne.ca/generating-sample-config-files-in-openstack/

You can see that the process isn't streamlined. Each project has its own
particularities. Some projects don't use the (un)official standard "tox
-egenconfig" command, I patched some projects to make it less a pain.

And my thoughts about sample config files:
http://blog.mgagne.ca/where-are-the-sample-config-files/

I wrote it after Cinder proposed removing its sample config file. They
abandoned the patch at that time but now sample config file is gone.

It's not an easy problem because core libraries used by OpenStack
projects also use oslo.config and configs required by those libraries
are part of the main configurations required for a project to even work.
([database]/connection for instance) You just can't ignore those configs
when generating the sample config file.

(sorry for self-promotion, I didn't want to rewrite my thoughts on this
list)

--
Mathieu


OpenStack-operators mailing list
OpenStack-operators at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

OpenStack-operators mailing list
OpenStack-operators at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

-------------- next part --------------
An HTML attachment was scrubbed...
URL:

responded Dec 10, 2014 by Anne_Gentle (6,980 points)   1 3 4
0 votes

From: Anne Gentle >
Date: Wednesday, December 10, 2014 at 8:41 AM
To: Michael Dorman >
Cc: "openstack-operators at lists.openstack.org" >
Subject: Re: [Openstack-operators] Packaging sample config versions

On Tue, Dec 9, 2014 at 11:25 AM, Michael Dorman > wrote:
Well I think we can all agree this is an irritation. But how are others
actually dealing with this problem? (Maybe it?s less complicated in
Ubuntu.)

The sense I get is that most people using Anvil, or other custom-ish
packaging tools, are also running config management which handles
generating the config files, anyway. So you don?t so much care about the
contents of the config file shipped with the package.

Is that accurate for most people? Or are folks doing some other magic to
get a good config file in the packages?

The docs team -- reallly, Matt Kassawara -- regularly logs bugs for packagers to put in better, working, default config files.

We do generate documentation for all the configurations across projects that use oslo.config (and even for swift, which doesn't). So you can rely on this reference:
http://docs.openstack.org/juno/config-reference/content/

I don?t see full sample configs in here, just more focused ?example? configs, am I missing it? Does the generated content include everything from config.py in each project? I?m asking because I?ve seen a few places where the documented defaults and whats in the code do not match.

When I upgrade puppet versions, like I?m doing now from the icehouse puppet branches, its very important for me to know:

  1. what has a new default value
  2. whether we were previously using the old default value
  3. what lines change/move/got renamed
  4. what is deprecated

The docs do a good job of most of these, but I cannot beat a well maintained full sample config.

Here are a few small examples of places where the docs and code didn?t match for Juno, hence my questions about how the docs were generated:

https://bugs.launchpad.net/openstack-manuals/+bug/1380778
https://bugs.launchpad.net/openstack-manuals/+bug/1380813

Note: I also filed two bugs against keystone itself for having default values that were deprecated, so its not perfect either.

You can also see new, updated, and deprecated options for each service, such as:
http://docs.openstack.org/juno/config-reference/content/nova-conf-changes-juno.html

I don't believe our reference document was what encouraged devs to take the sample config generation out-of-tree, but I am letting you know your best option besides troubleshooting generating it yourself.

Anne

Mike

On 12/9/14, 5:02 PM, "Kris G. Lindgren" > wrote:

So more to my point on the latest version of RHEL and doing: yum install
tox -egenconfig

ceilometer-2014.2.1]# tox -egenconfig
ERROR: tox version is 1.4.2, required is at least 1.6

nova-2014.2.1]# tox -egenconfig
ERROR: tox version is 1.4.2, required is at least 1.6

glance-2014.2.1]# tox -egenconfig
ERROR: tox version is 1.4.2, required is at least 1.6

[root at localhost ~]# pip install --update tox
(Updated tox to 1.8.1 , upgraded virtualenv to 1.10.1 and upgraded py to
1.4.14)

glance-2014.2.1]# tox -egenconfig
genconfig create: /root/rpmbuild/BUILD/glance-2014.2.1/.tox/genconfig
genconfig installdeps:
-r/root/rpmbuild/BUILD/glance-2014.2.1/requirements.txt,
-r/root/rpmbuild/BUILD/glance-2014.2.1/test-requirements.txt
ERROR: invocation failed (exit code 1), logfile:
/root/rpmbuild/BUILD/glance-2014.2.1/.tox/genconfig/log/genconfig-1.log
ERROR: actionid=genconfig

Running setup.py install for MySQL-python

/usr/include/mysql/myconfigx8664.h:654:2: error: #error
<my
config.h> MUST be included first!
#error <my_config.h> MUST be included first!
^
error: command 'gcc' failed with exit status 1

__________________________________________________________________ summary
ERROR: genconfig: could not install deps
[-r/root/rpmbuild/BUILD/glance-2014.2.1/requirements.txt,
-r/root/rpmbuild/BUILD/glance-2014.2.1/test-requirements.txt]; v =
InvocationError('/root/rpmbuild/BUILD/glance-2014.2.1/.tox/genconfig/bin/p
i
p install --allow-all-external --allow-insecure netaddr -U
-r/root/rpmbuild/BUILD/glance-2014.2.1/requirements.txt
-r/root/rpmbuild/BUILD/glance-2014.2.1/test-requirements.txt (see
/root/rpmbuild/BUILD/glance-2014.2.1/.tox/genconfig/log/genconfig-1.log)',
1)

So a few things to point out in order to even get tox -egenconfig I had to
update the system packages versions using pip. Since we have other python
packages using virtualenv I have no idea if the updated venvironment
package is going to break those systems or not. So the included
script/command is already a barrier to getting a sample config. 2) tox
fails to even build all the deps - it happens to be exactly failing at
mysql in both nova/cinder/glance/keystone 3) It's installing it own
versions of python libraries that solve the dependencies that are then
going to be used to generate the configuration. If the configuration is
so dynamic that getting a different version of oslo.config could generate
a sample configuration that wont work on my system then how am I suppose
to deal with:
Tox installed version:
oslo.config-1.5.0

System installed version:
python-oslo-config-1.3.0

Also python-libvrit failed to build because I don?t have libvrit installed
on this system. So am I to assume that there are no libvrit options
(which we both know is false)?
Now I can get a example config - that wont work with my system - per what
everyone else has been saying. Also, at what point would the average user
just say "F it"? - because at the point I feel like if I was an average
user - I would be there right now.


Kris Lindgren
Senior Linux Systems Engineer
GoDaddy, LLC.

On 12/9/14, 8:14 AM, "Mathieu Gagn?" > wrote:

On 2014-12-08 11:01 PM, Kris G. Lindgren wrote:

I don?t think its too much to ask for each project to include a
script
that will build a venv that includes tox and the other relevant deps to
build the sample configuration.

This is already the case. Back then, I did the work of documenting how
you could generate the sample config files for each projects (I cared
about):
http://blog.mgagne.ca/generating-sample-config-files-in-openstack/

You can see that the process isn't streamlined. Each project has its own
particularities. Some projects don't use the (un)official standard "tox
-egenconfig" command, I patched some projects to make it less a pain.

And my thoughts about sample config files:
http://blog.mgagne.ca/where-are-the-sample-config-files/

I wrote it after Cinder proposed removing its sample config file. They
abandoned the patch at that time but now sample config file is gone.

It's not an easy problem because core libraries used by OpenStack
projects also use oslo.config and configs required by those libraries
are part of the main configurations required for a project to even work.
([database]/connection for instance) You just can't ignore those configs
when generating the sample config file.

(sorry for self-promotion, I didn't want to rewrite my thoughts on this
list)

--
Mathieu


OpenStack-operators mailing list
OpenStack-operators at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

OpenStack-operators mailing list
OpenStack-operators at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


This E-mail and any of its attachments may contain Time Warner Cable proprietary information, which is privileged, confidential, or subject to copyright belonging to Time Warner Cable. This E-mail is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient of this E-mail, you are hereby notified that any dissemination, distribution, copying, or action taken in relation to the contents of and attachments to this E-mail is strictly prohibited and may be unlawful. If you have received this E-mail in error, please notify the sender immediately and permanently delete the original and any copy of this E-mail and any printout.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
responded Dec 10, 2014 by Fischer,_Matt (1,380 points)   1 3
...