settingsLogin | Registersettings

[openstack-dev] [packaging][all] Sample Config Files in setup.cfg

0 votes

There’s some history around this discussion [1], but times have changed and the purpose of the patches I’m submitting is slightly different [2] as far as I can see – it’s a little more focused and less intrusive.

The projects which deploy OpenStack from source or using python wheels currently have to either carry templates for api-paste, policy and rootwrap files or need to source them from git during deployment. This results in some rather complex mechanisms which could be radically simplified by simply ensuring that all the same files are included in the built wheel. Distribution packagers typically also have mechanisms in place to fetch the files from the source repo when building the packages – including the files through pbr’s data_files for packagers may or may not be beneficial, depending on how the packagers do their build processes.

In neutron [3], glance [4], designate [5] and sahara [6] the use of the data_files option in the files section of setup.cfg is established and has been that way for some time. However, there have been issues in the past implementing something similar – for example in keystone there has been a bit of a yoyo situation where a patch was submitted, then reverted.

I’ve been proposing patches [7] to try to make the implementation across projects consistent and projects have, for the most part, been happy to go ahead and merge them. However concern has been raised that we may end up going through another yo-yo experience and therefore I’ve been asked to raise this on the ML.

Do any packagers or deployment projects have issues with this implementation? If there are any issues, what’re your suggestions to resolve them?

[1] http://lists.openstack.org/pipermail/openstack-dev/2016-June/097123.html
[2] https://launchpad.net/bugs/1718356
[3] https://github.com/openstack/neutron/blob/d3c393ff6b5fbd0bdaabc8ba678d755ebfba08f7/setup.cfg#L24-L39
[4] https://github.com/openstack/glance/blob/02cd5cba70a8465a951cb813a573d390887174b7/setup.cfg#L20-L21
[5] https://github.com/openstack/designate/blob/25eb143db04554d65efe2e5d60ad3afa6b51d73a/setup.cfg#L30-L37
[6] https://github.com/openstack/sahara/blob/cff43d6f1eee5c68af16c6f655f4d019669224d9/setup.cfg#L28-L29
[7] https://review.openstack.org/#/q/topic:bug/1718356+(status:open+OR+status:merged)


Rackspace Limited is a company registered in England & Wales (company registered number 03897010) whose registered office is at 5 Millington Road, Hyde Park Hayes, Middlesex UB3 4AZ. Rackspace Limited privacy policy can be viewed at www.rackspace.co.uk/legal/privacy-policy - This e-mail message may contain confidential or privileged information intended for the recipient. Any dissemination, distribution or copying of the enclosed material is prohibited. If you receive this transmission in error, please notify us immediately by e-mail at abuse@rackspace.com and delete the original message. Your cooperation is appreciated.

OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Oct 12, 2017 in openstack-dev by Jesse.Pretorius_at_r (2,260 points)   1 2

39 Responses

0 votes

Excerpts from Jesse Pretorius's message of 2017-09-28 14:50:24 +0000:

There’s some history around this discussion [1], but times have changed and the purpose of the patches I’m submitting is slightly different [2] as far as I can see – it’s a little more focused and less intrusive.

The projects which deploy OpenStack from source or using python wheels currently have to either carry templates for api-paste, policy and rootwrap files or need to source them from git during deployment. This results in some rather complex mechanisms which could be radically simplified by simply ensuring that all the same files are included in the built wheel. Distribution packagers typically also have mechanisms in place to fetch the files from the source repo when building the packages – including the files through pbr’s data_files for packagers may or may not be beneficial, depending on how the packagers do their build processes.

In neutron [3], glance [4], designate [5] and sahara [6] the use of the data_files option in the files section of setup.cfg is established and has been that way for some time. However, there have been issues in the past implementing something similar – for example in keystone there has been a bit of a yoyo situation where a patch was submitted, then reverted.

I’ve been proposing patches [7] to try to make the implementation across projects consistent and projects have, for the most part, been happy to go ahead and merge them. However concern has been raised that we may end up going through another yo-yo experience and therefore I’ve been asked to raise this on the ML.

Do any packagers or deployment projects have issues with this implementation? If there are any issues, what’re your suggestions to resolve them?

[1] http://lists.openstack.org/pipermail/openstack-dev/2016-June/097123.html
[2] https://launchpad.net/bugs/1718356
[3] https://github.com/openstack/neutron/blob/d3c393ff6b5fbd0bdaabc8ba678d755ebfba08f7/setup.cfg#L24-L39
[4] https://github.com/openstack/glance/blob/02cd5cba70a8465a951cb813a573d390887174b7/setup.cfg#L20-L21
[5] https://github.com/openstack/designate/blob/25eb143db04554d65efe2e5d60ad3afa6b51d73a/setup.cfg#L30-L37
[6] https://github.com/openstack/sahara/blob/cff43d6f1eee5c68af16c6f655f4d019669224d9/setup.cfg#L28-L29
[7] https://review.openstack.org/#/q/topic:bug/1718356+(status:open+OR+status:merged)

In the past we had trouble checking those files into git and gating
against the results being "up to date" or not changing in any way
because configuration options that end up in the file are defined in
libraries used by the services. So as long as the implementation you're
considering does not check configuration files into git, but generates
them and then inserts them into the package, it should be fine.

Doug


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 28, 2017 by Doug_Hellmann (87,520 points)   3 4 5
0 votes

On 9/28/17, 5:11 PM, "Doug Hellmann" doug@doughellmann.com wrote:

In the past we had trouble checking those files into git and gating
against the results being "up to date" or not changing in any way
because configuration options that end up in the file are defined in
libraries used by the services. So as long as the implementation you're
considering does not check configuration files into git, but generates
them and then inserts them into the package, it should be fine.

I’m guessing that the aut-generated files you’re referring to are the .conf files? For the most part, those are being left out of my proposed patches unless the project team specifically requests their inclusion. My patches are focused on the far more static files - policy.json if it exists (yes, the policy-in-code will remove those in time), api-paste, rootwrap.conf and the rootwrap.d contents. As far as I know none of these are auto-generated. If they are, I’m all ears to learn how!


Rackspace Limited is a company registered in England & Wales (company registered number 03897010) whose registered office is at 5 Millington Road, Hyde Park Hayes, Middlesex UB3 4AZ. Rackspace Limited privacy policy can be viewed at www.rackspace.co.uk/legal/privacy-policy - This e-mail message may contain confidential or privileged information intended for the recipient. Any dissemination, distribution or copying of the enclosed material is prohibited. If you receive this transmission in error, please notify us immediately by e-mail at abuse@rackspace.com and delete the original message. Your cooperation is appreciated.

OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 28, 2017 by Jesse.Pretorius_at_r (2,260 points)   1 2
0 votes

Excerpts from Jesse Pretorius's message of 2017-09-28 17:17:55 +0000:

On 9/28/17, 5:11 PM, "Doug Hellmann" doug@doughellmann.com wrote:

In the past we had trouble checking those files into git and gating
against the results being "up to date" or not changing in any way
because configuration options that end up in the file are defined in
libraries used by the services. So as long as the implementation you're
considering does not check configuration files into git, but generates
them and then inserts them into the package, it should be fine.

I’m guessing that the aut-generated files you’re referring to are the .conf files? For the most part, those are being left out of my proposed patches unless the project team specifically requests their inclusion. My patches are focused on the far more static files - policy.json if it exists (yes, the policy-in-code will remove those in time), api-paste, rootwrap.conf and the rootwrap.d contents. As far as I know none of these are auto-generated. If they are, I’m all ears to learn how!

Ah, yes, I was talking about oslo.config files but those other files
are more static and shouldn't present the same issue.

Doug


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 28, 2017 by Doug_Hellmann (87,520 points)   3 4 5
0 votes

Hi,

On 28.09.2017 16:50, Jesse Pretorius wrote:
[...]
Do any packagers or deployment projects have issues with this
implementation? If there are any issues, what’re your suggestions to
resolve them?

This will still install the files into usr/etc :

$ python setup.py install --skip-build --root /tmp/sahara-install >
/dev/null
$ ls /tmp/sahara-install/usr/
bin etc lib

It's not nice but packagers can workaround that.

Best,

Tom


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 29, 2017 by tbechtold_at_suse.co (1,680 points)   2
0 votes

On 9/29/17, 7:18 AM, "Thomas Bechtold" tbechtold@suse.com wrote:

This will still install the files into usr/etc :
It's not nice but packagers can workaround that.

Yes, that is true. Is there a ‘better’ location to have them? I noticed that Sahara was placing the files into share, resulting in them being installed into /usr/share – is that better?

For OSA as a project it’s not really a problem where it goes, just that the files are there and ideally in a consistent place.


Rackspace Limited is a company registered in England & Wales (company registered number 03897010) whose registered office is at 5 Millington Road, Hyde Park Hayes, Middlesex UB3 4AZ. Rackspace Limited privacy policy can be viewed at www.rackspace.co.uk/legal/privacy-policy - This e-mail message may contain confidential or privileged information intended for the recipient. Any dissemination, distribution or copying of the enclosed material is prohibited. If you receive this transmission in error, please notify us immediately by e-mail at abuse@rackspace.com and delete the original message. Your cooperation is appreciated.

OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 29, 2017 by Jesse.Pretorius_at_r (2,260 points)   1 2
0 votes

hi,

On 29.09.2017 12:56, Jesse Pretorius wrote:
On 9/29/17, 7:18 AM, "Thomas Bechtold" tbechtold@suse.com wrote:

 This will still install the files into usr/etc :
 It's not nice but packagers can workaround that.

Yes, that is true. Is there a ‘better’ location to have them? I noticed that Sahara was placing the files into share, resulting in them being installed into /usr/share – is that better?

There is /etc [1]

Best,

Tom

[1]
http://refspecs.linuxfoundation.org/FHS_3.0/fhs-3.0.html#etcHostspecificSystemConfiguration


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 29, 2017 by tbechtold_at_suse.co (1,680 points)   2
0 votes

On 2017-09-29 18:39:18 +0200 (+0200), Thomas Bechtold wrote:
On 29.09.2017 12:56, Jesse Pretorius wrote:

On 9/29/17, 7:18 AM, "Thomas Bechtold" tbechtold@suse.com wrote:

This will still install the files into usr/etc :
It's not nice but packagers can workaround that.

Yes, that is true. Is there a ‘better’ location to have them? I
noticed that Sahara was placing the files into share, resulting
in them being installed into /usr/share – is that better?

There is /etc [1]
[...]

Not really, no, because the system-context data_files path has to be
relative to /usr or /usr/local unless we want to have modules going
into /lib and entrypoints in /bin now. There was a somewhat thorough
discussion on the distutils-sig ML a couple years ago (I think? my
sense of time is pretty terrible) of a potential solution involving
setting up a taxonomy of data file types and allowing
installers/package maintainers to map those to whatever they want
(FHS for distro packages maybe, relative to the install root for
virtualenvs, and so on). I can't seem to find that proposal now, nor
does it appear to have gotten as far as becoming a PEP, but I'll
admit I only spent a few minutes digging around for it.

The current situation with data_files behavior is complicated by the
multitude of ways and places Python packages can be installed,
resulting in bug reports like:

https://github.com/pypa/setuptools/issues/130
https://github.com/pypa/setuptools/issues/460
https://github.com/pypa/setuptools/issues/807

Also there's the problem that it's just a single bucket which could
be used to house example configuration files, shared datasets,
manpages or similar documentation... so dumping all those into a
single path is pretty messy anyway.
--
Jeremy Stanley


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

responded Sep 29, 2017 by Jeremy_Stanley (56,700 points)   3 5 7
0 votes

On 9/29/17, 6:26 PM, "Jeremy Stanley" fungi@yuggoth.org wrote:

On 2017-09-29 18:39:18 +0200 (+0200), Thomas Bechtold wrote:

There is /etc [1]
[...]

Not really, no, because the system-context data_files path has to be
relative to /usr or /usr/local unless we want to have modules going
into /lib and entrypoints in /bin now.

Right – that’s exactly why I this it would be better to stick with a relative path, but make the implementation consistent.

So, given a relative path being used – which is better: etc or share?

To me, etc seems more intuitive given that these are configuration files. Using etc benefits those building and consuming wheels by being an intuitive placement (putting the files into the etc directory of the venv). Each packaging system has their own conventions so I do not think we’re going to be able to come to a common consensus that pleases everyone, so I’d like to rather focus on attaining a consistent path across services so that packagers can adapt their scripts appropriately to cater for their individual quirks, while everyone using wheels gets the benefit of the files being a part of the package.


Rackspace Limited is a company registered in England & Wales (company registered number 03897010) whose registered office is at 5 Millington Road, Hyde Park Hayes, Middlesex UB3 4AZ. Rackspace Limited privacy policy can be viewed at www.rackspace.co.uk/legal/privacy-policy - This e-mail message may contain confidential or privileged information intended for the recipient. Any dissemination, distribution or copying of the enclosed material is prohibited. If you receive this transmission in error, please notify us immediately by e-mail at abuse@rackspace.com and delete the original message. Your cooperation is appreciated.

OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 2, 2017 by Jesse.Pretorius_at_r (2,260 points)   1 2
0 votes

On 09/28/2017 04:50 PM, Jesse Pretorius wrote:
There’s some history around this discussion [1], but times have changed
and the purpose of the patches I’m submitting is slightly different [2]
as far as I can see – it’s a little more focused and less intrusive.

 

The projects which deploy OpenStack from source or using python wheels
currently have to either carry templates for api-paste, policy and
rootwrap files or need to source them from git during deployment. This
results in some rather complex mechanisms which could be radically
simplified by simply ensuring that all the same files are included in
the built wheel. Distribution packagers typically also have mechanisms
in place to fetch the files from the source repo when building the
packages – including the files through pbr’s data_files for packagers
may or may not be beneficial, depending on how the packagers do their
build processes.

 

In neutron [3], glance [4], designate [5] and sahara [6] the use of the
data_files option in the files section of setup.cfg is established and
has been that way for some time. However, there have been issues in the
past implementing something similar – for example in keystone there has
been a bit of a yoyo situation where a patch was submitted, then reverted.

 

I’ve been proposing patches [7] to try to make the implementation across
projects consistent and projects have, for the most part, been happy to
go ahead and merge them. However concern has been raised that we may end
up going through another yo-yo experience and therefore I’ve been asked
to raise this on the ML.

 

Do any packagers or deployment projects have issues with this
implementation? If there are any issues, what’re your suggestions to
resolve them?

I still have the issue that adding stuff in etc, at packaging time, push
them in /usr/etc, which is obviously wrong. We tried to push for a PBR
patch, but failed, as Robert Collins wrote it had to be fixed in
setuptools. Which is why all patches have been reverted so far.

While I may agree with Robert, I think we had to choose for a pragmatic
(even temporary) solution, and I don't understand why it's been years
that this stays unfixed, especially when we have an easy solution. [1]

So, until that is fixed, please do not propose this type of patches.

Cheers,

Thomas Goirand (zigo)

[1] https://review.openstack.org/#/c/274077/


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 2, 2017 by Thomas_Goirand (18,640 points)   2 9 16
0 votes

On Monday, 2 October 2017 13:28:17 CEST Thomas Goirand wrote:
On 09/28/2017 04:50 PM, Jesse Pretorius wrote:

There’s some history around this discussion [1], but times have changed
and the purpose of the patches I’m submitting is slightly different [2]
as far as I can see – it’s a little more focused and less intrusive.

The projects which deploy OpenStack from source or using python wheels
currently have to either carry templates for api-paste, policy and
rootwrap files or need to source them from git during deployment. This
results in some rather complex mechanisms which could be radically
simplified by simply ensuring that all the same files are included in
the built wheel. Distribution packagers typically also have mechanisms
in place to fetch the files from the source repo when building the
packages – including the files through pbr’s data_files for packagers
may or may not be beneficial, depending on how the packagers do their
build processes.

In neutron [3], glance [4], designate [5] and sahara [6] the use of the
data_files option in the files section of setup.cfg is established and
has been that way for some time. However, there have been issues in the
past implementing something similar – for example in keystone there has
been a bit of a yoyo situation where a patch was submitted, then reverted.

I’ve been proposing patches [7] to try to make the implementation across
projects consistent and projects have, for the most part, been happy to
go ahead and merge them. However concern has been raised that we may end
up going through another yo-yo experience and therefore I’ve been asked
to raise this on the ML.

Do any packagers or deployment projects have issues with this
implementation? If there are any issues, what’re your suggestions to
resolve them?

I still have the issue that adding stuff in etc, at packaging time, push
them in /usr/etc, which is obviously wrong. We tried to push for a PBR
patch, but failed, as Robert Collins wrote it had to be fixed in
setuptools. Which is why all patches have been reverted so far.

While I may agree with Robert, I think we had to choose for a pragmatic
(even temporary) solution, and I don't understand why it's been years
that this stays unfixed, especially when we have an easy solution. [1]

So, until that is fixed, please do not propose this type of patches.

Why not? Even if it does not fix the issue for proper installations,
- it does not provent people from copying the files somewhere else (it
happened in sahara for how long I can remember, we have been using datafiles)
- it fixes the deployment when the package is installed in a virtualenv;
- it introduces consistency: the day data
files starts to do the right thing,
everything will work; if it's not possible to fix it with datafiles, it's
easy to spot which files should be fixed because all handled by data
files.

So definitely go for it.

Ciao
--
Luigi


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 2, 2017 by Luigi_Toscano (1,700 points)   2
...