settingsLogin | Registersettings

[openstack-dev] [neutron][puppet] mod_wsgi support (pike bug?)

0 votes

On Mon, Sep 4, 2017 at 9:01 AM, Kevin Benton kevin@benton.pub wrote:

Yes, unfortunately I didn't make it back to the patch in time to adjust
devstack to dump all of the configuration into one file (instead of
/etc/neutron/neutron.conf /etc/neutron/plugins/ml2.conf etc). I did test
locally with my dev environment around the time that RPC server patch went
in, but there may have been a regression since it went in since it's not
tested as Matt pointed out.

I've added Puppet into this because I think we would have to take a
decision regarding this. The reason behind the fact that we've always
used the two configuration files is because distributions which ship
packages actually provide 2 configuration files.

We use a configuration resource called neutron_plugin_ml2 which
configures things always in /etc/neutron/plugins/ml2/ml2_conf.ini.

In the case of Ubuntu/Debian based systems, we update
/etc/default/neutron-server to point the plugin configuration to
/etc/neutron/plugin.ini and then we ensure that the file is a
symbolic link to /etc/neutron/plugins/ml2/ml2_conf.ini.

Now, we have two options in my mind (and I am open for others):

1) Configure plugins entirely inside neutron.conf
This option would solve our issue but introduce another one. I
believe that the order in the start commands would mean that the later
configuration files (in this case, the plugin.ini) would have priority
over the neutron.conf causing an inconsistency, I don't think this
is possible. However, we can work around this by ensuring that the
plugin.ini file is empty. However, we will be introducing service
restarts for no reason with that change which can be very confusing
for the user as to why configuration is moving.

2) Figure out a way to pass 'args' via WSGI?
I'm not sure if this is entirely possible at all. But, could there be
a way that we can pass a list of configuration files in the mod_wsgi
configuration? This would make it the most transparent fix without
having to adjust all of the configuration files and bend against the
set configuration paths of the distro.

I'd be more than happy to hear any other ideas regarding this
solution. I would love to implement #2 if it is somehow possible
(environment variables, maybe?) but #1 would work but be very messy
for operators and users.

It appears that puppet is still spreading the config files for the server
into multiple locations as well[1]. Does it inherit that logic from
devstack? Because that will need to be changed to push all of the relevant
server config into one conf.

1.
http://logs.openstack.org/82/500182/3/check/gate-puppet-openstack-integration-4-scenario004-tempest-ubuntu-xenial/791523c/logs/etc/neutron/plugins/

On Sun, Sep 3, 2017 at 12:03 PM, Mohammed Naser mnaser@vexxhost.com wrote:

On Sun, Sep 3, 2017 at 3:03 PM, Mohammed Naser mnaser@vexxhost.com
wrote:

On Sun, Sep 3, 2017 at 2:20 PM, Matthew Treinish mtreinish@kortar.org
wrote:

On Sun, Sep 03, 2017 at 01:47:24PM -0400, Mohammed Naser wrote:

Hi folks,

I've attempted to enable modwsgi support in our dev environment with
Puppet however it results in a traceback. I figured it was an
environment thing so I looked into moving the Puppet CI to test using
mod
wsgi and it resulted in the same error.

http://logs.openstack.org/82/500182/3/check/gate-puppet-openstack-integration-4-scenario004-tempest-ubuntu-xenial/791523c/logs/apache/neutron_wsgi_error.txt.gz

Would anyone from the Neutron team be able to give input on this?
We'd love to add gating for Neutron deployed by mod_wsgi which can
help find similar issues.

Neutron never got their wsgi support working in Devstack either. The
patch
adding that: https://review.openstack.org/#/c/439191/ never passed the
gate and
seems to have lost the attention of the author. The wsgi support in
neutron
probably doesn't work yet, and is definitely untested. IIRC, the issue
they were
hitting was loading the config files. [1] I don't think I saw any
progress on it
after that though.

The TC goal doc [2] probably should say something about it never
landing and
missing pike.

That would make sense. The release notes also state that it does
offer the ability to run inside mod_wsgi which can be misleading to
deployers (that was the main reason I thought we can start testing
using it):

Sigh, hit send too early. Here is the link:

http://git.openstack.org/cgit/openstack/neutron/commit/?id=916bc96ee214078496b4b38e1c93f36f906ce840

-Matt Treinish

[1]
http://lists.openstack.org/pipermail/openstack-dev/2017-June/117830.html
[2]
https://governance.openstack.org/tc/goals/pike/deploy-api-in-wsgi.html#neutron


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Sep 4, 2017 in openstack-dev by Mohammed_Naser (3,860 points)   1 3

7 Responses

0 votes

On Mon, Sep 4, 2017 at 10:40 AM, Mohammed Naser mnaser@vexxhost.com wrote:
On Mon, Sep 4, 2017 at 9:01 AM, Kevin Benton kevin@benton.pub wrote:

Yes, unfortunately I didn't make it back to the patch in time to adjust
devstack to dump all of the configuration into one file (instead of
/etc/neutron/neutron.conf /etc/neutron/plugins/ml2.conf etc). I did test
locally with my dev environment around the time that RPC server patch went
in, but there may have been a regression since it went in since it's not
tested as Matt pointed out.

I've added Puppet into this because I think we would have to take a
decision regarding this. The reason behind the fact that we've always
used the two configuration files is because distributions which ship
packages actually provide 2 configuration files.

We use a configuration resource called neutron_plugin_ml2 which
configures things always in /etc/neutron/plugins/ml2/ml2_conf.ini.

In the case of Ubuntu/Debian based systems, we update
/etc/default/neutron-server to point the plugin configuration to
/etc/neutron/plugin.ini and then we ensure that the file is a
symbolic link to /etc/neutron/plugins/ml2/ml2_conf.ini.

Now, we have two options in my mind (and I am open for others):

1) Configure plugins entirely inside neutron.conf
This option would solve our issue but introduce another one. I
believe that the order in the start commands would mean that the later
configuration files (in this case, the plugin.ini) would have priority
over the neutron.conf causing an inconsistency, I don't think this
is possible. However, we can work around this by ensuring that the
plugin.ini file is empty. However, we will be introducing service
restarts for no reason with that change which can be very confusing
for the user as to why configuration is moving.

2) Figure out a way to pass 'args' via WSGI?
I'm not sure if this is entirely possible at all. But, could there be
a way that we can pass a list of configuration files in the mod_wsgi
configuration? This would make it the most transparent fix without
having to adjust all of the configuration files and bend against the
set configuration paths of the distro.

I did a bit more research and unfortunately it looks like this option
is not possible (at least with mod_wsgi):

https://github.com/GrahamDumpleton/mod_wsgi/blob/master/src/server/wsgi_interp.c#L546-L561

modwsgi sets the content of sys.argv to ["modwsgi"] only.
Environment variables are still a possibility.

I'd be more than happy to hear any other ideas regarding this
solution. I would love to implement #2 if it is somehow possible
(environment variables, maybe?) but #1 would work but be very messy
for operators and users.

It appears that puppet is still spreading the config files for the server
into multiple locations as well[1]. Does it inherit that logic from
devstack? Because that will need to be changed to push all of the relevant
server config into one conf.

1.
http://logs.openstack.org/82/500182/3/check/gate-puppet-openstack-integration-4-scenario004-tempest-ubuntu-xenial/791523c/logs/etc/neutron/plugins/

On Sun, Sep 3, 2017 at 12:03 PM, Mohammed Naser mnaser@vexxhost.com wrote:

On Sun, Sep 3, 2017 at 3:03 PM, Mohammed Naser mnaser@vexxhost.com
wrote:

On Sun, Sep 3, 2017 at 2:20 PM, Matthew Treinish mtreinish@kortar.org
wrote:

On Sun, Sep 03, 2017 at 01:47:24PM -0400, Mohammed Naser wrote:

Hi folks,

I've attempted to enable modwsgi support in our dev environment with
Puppet however it results in a traceback. I figured it was an
environment thing so I looked into moving the Puppet CI to test using
mod
wsgi and it resulted in the same error.

http://logs.openstack.org/82/500182/3/check/gate-puppet-openstack-integration-4-scenario004-tempest-ubuntu-xenial/791523c/logs/apache/neutron_wsgi_error.txt.gz

Would anyone from the Neutron team be able to give input on this?
We'd love to add gating for Neutron deployed by mod_wsgi which can
help find similar issues.

Neutron never got their wsgi support working in Devstack either. The
patch
adding that: https://review.openstack.org/#/c/439191/ never passed the
gate and
seems to have lost the attention of the author. The wsgi support in
neutron
probably doesn't work yet, and is definitely untested. IIRC, the issue
they were
hitting was loading the config files. [1] I don't think I saw any
progress on it
after that though.

The TC goal doc [2] probably should say something about it never
landing and
missing pike.

That would make sense. The release notes also state that it does
offer the ability to run inside mod_wsgi which can be misleading to
deployers (that was the main reason I thought we can start testing
using it):

Sigh, hit send too early. Here is the link:

http://git.openstack.org/cgit/openstack/neutron/commit/?id=916bc96ee214078496b4b38e1c93f36f906ce840

-Matt Treinish

[1]
http://lists.openstack.org/pipermail/openstack-dev/2017-June/117830.html
[2]
https://governance.openstack.org/tc/goals/pike/deploy-api-in-wsgi.html#neutron


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 4, 2017 by Mohammed_Naser (3,860 points)   1 3
0 votes

2 would be preferable as well just because we have so many guides/distros

mentioning the different file locations. I'm not familiar with mod_wsgi
enough to know if it's possible.

Another 3rd option may be to edit the oslo config constructor call when
using that entry point to add some well-known paths for additional config
files rather than only parsing 'sys.argv[1]'. For example, we could always
try to add /etc/neutron/plugin.ini to the list which we can document that
distros/deployment tools should always create or have a symbolic link to a
plugin-specific config (e.g. ml2_conf.ini).

On Mon, Sep 4, 2017 at 7:40 AM, Mohammed Naser mnaser@vexxhost.com wrote:

On Mon, Sep 4, 2017 at 9:01 AM, Kevin Benton kevin@benton.pub wrote:

Yes, unfortunately I didn't make it back to the patch in time to adjust
devstack to dump all of the configuration into one file (instead of
/etc/neutron/neutron.conf /etc/neutron/plugins/ml2.conf etc). I did test
locally with my dev environment around the time that RPC server patch
went
in, but there may have been a regression since it went in since it's not
tested as Matt pointed out.

I've added Puppet into this because I think we would have to take a
decision regarding this. The reason behind the fact that we've always
used the two configuration files is because distributions which ship
packages actually provide 2 configuration files.

We use a configuration resource called neutron_plugin_ml2 which
configures things always in /etc/neutron/plugins/ml2/ml2_conf.ini.

In the case of Ubuntu/Debian based systems, we update
/etc/default/neutron-server to point the plugin configuration to
/etc/neutron/plugin.ini and then we ensure that the file is a
symbolic link to /etc/neutron/plugins/ml2/ml2_conf.ini.

Now, we have two options in my mind (and I am open for others):

1) Configure plugins entirely inside neutron.conf
This option would solve our issue but introduce another one. I
believe that the order in the start commands would mean that the later
configuration files (in this case, the plugin.ini) would have priority
over the neutron.conf causing an inconsistency, I don't think this
is possible. However, we can work around this by ensuring that the
plugin.ini file is empty. However, we will be introducing service
restarts for no reason with that change which can be very confusing
for the user as to why configuration is moving.

2) Figure out a way to pass 'args' via WSGI?
I'm not sure if this is entirely possible at all. But, could there be
a way that we can pass a list of configuration files in the mod_wsgi
configuration? This would make it the most transparent fix without
having to adjust all of the configuration files and bend against the
set configuration paths of the distro.

I'd be more than happy to hear any other ideas regarding this
solution. I would love to implement #2 if it is somehow possible
(environment variables, maybe?) but #1 would work but be very messy
for operators and users.

It appears that puppet is still spreading the config files for the server
into multiple locations as well[1]. Does it inherit that logic from
devstack? Because that will need to be changed to push all of the
relevant
server config into one conf.

1.
http://logs.openstack.org/82/500182/3/check/gate-puppet-
openstack-integration-4-scenario004-tempest-ubuntu-
xenial/791523c/logs/etc/neutron/plugins/

On Sun, Sep 3, 2017 at 12:03 PM, Mohammed Naser mnaser@vexxhost.com
wrote:

On Sun, Sep 3, 2017 at 3:03 PM, Mohammed Naser mnaser@vexxhost.com
wrote:

On Sun, Sep 3, 2017 at 2:20 PM, Matthew Treinish <
mtreinish@kortar.org>
wrote:

On Sun, Sep 03, 2017 at 01:47:24PM -0400, Mohammed Naser wrote:

Hi folks,

I've attempted to enable modwsgi support in our dev environment
with
Puppet however it results in a traceback. I figured it was an
environment thing so I looked into moving the Puppet CI to test
using
mod
wsgi and it resulted in the same error.

http://logs.openstack.org/82/500182/3/check/gate-puppet-
openstack-integration-4-scenario004-tempest-ubuntu-
xenial/791523c/logs/apache/neutronwsgierror.txt.gz

Would anyone from the Neutron team be able to give input on this?
We'd love to add gating for Neutron deployed by mod_wsgi which can
help find similar issues.

Neutron never got their wsgi support working in Devstack either. The
patch
adding that: https://review.openstack.org/#/c/439191/ never passed
the
gate and
seems to have lost the attention of the author. The wsgi support in
neutron
probably doesn't work yet, and is definitely untested. IIRC, the
issue
they were
hitting was loading the config files. [1] I don't think I saw any
progress on it
after that though.

The TC goal doc [2] probably should say something about it never
landing and
missing pike.

That would make sense. The release notes also state that it does
offer the ability to run inside mod_wsgi which can be misleading to
deployers (that was the main reason I thought we can start testing
using it):

Sigh, hit send too early. Here is the link:

http://git.openstack.org/cgit/openstack/neutron/commit/?id=
916bc96ee214078496b4b38e1c93f36f906ce840

-Matt Treinish

[1]
http://lists.openstack.org/pipermail/openstack-dev/2017-
June/117830.html
[2]
https://governance.openstack.org/tc/goals/pike/deploy-api-
in-wsgi.html#neutron



OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:
unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:
unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 4, 2017 by kevin_at_benton.pub (15,600 points)   2 3 4
0 votes

Expanding on my other email. We could alter the oslo.config constructor
call to pass in file paths from environmental variables as well. Maybe
that's the easiest path forward since it would just be a change from
sys.argv to the env vars.

On Mon, Sep 4, 2017 at 8:04 AM, Mohammed Naser mnaser@vexxhost.com wrote:

On Mon, Sep 4, 2017 at 10:40 AM, Mohammed Naser mnaser@vexxhost.com
wrote:

On Mon, Sep 4, 2017 at 9:01 AM, Kevin Benton kevin@benton.pub wrote:

Yes, unfortunately I didn't make it back to the patch in time to adjust
devstack to dump all of the configuration into one file (instead of
/etc/neutron/neutron.conf /etc/neutron/plugins/ml2.conf etc). I did test
locally with my dev environment around the time that RPC server patch
went
in, but there may have been a regression since it went in since it's not
tested as Matt pointed out.

I've added Puppet into this because I think we would have to take a
decision regarding this. The reason behind the fact that we've always
used the two configuration files is because distributions which ship
packages actually provide 2 configuration files.

We use a configuration resource called neutron_plugin_ml2 which
configures things always in /etc/neutron/plugins/ml2/ml2_conf.ini.

In the case of Ubuntu/Debian based systems, we update
/etc/default/neutron-server to point the plugin configuration to
/etc/neutron/plugin.ini and then we ensure that the file is a
symbolic link to /etc/neutron/plugins/ml2/ml2_conf.ini.

Now, we have two options in my mind (and I am open for others):

1) Configure plugins entirely inside neutron.conf
This option would solve our issue but introduce another one. I
believe that the order in the start commands would mean that the later
configuration files (in this case, the plugin.ini) would have priority
over the neutron.conf causing an inconsistency, I don't think this
is possible. However, we can work around this by ensuring that the
plugin.ini file is empty. However, we will be introducing service
restarts for no reason with that change which can be very confusing
for the user as to why configuration is moving.

2) Figure out a way to pass 'args' via WSGI?
I'm not sure if this is entirely possible at all. But, could there be
a way that we can pass a list of configuration files in the mod_wsgi
configuration? This would make it the most transparent fix without
having to adjust all of the configuration files and bend against the
set configuration paths of the distro.

I did a bit more research and unfortunately it looks like this option
is not possible (at least with mod_wsgi):

https://github.com/GrahamDumpleton/mod_wsgi/blob/master/src/
server/wsgi_interp.c#L546-L561

modwsgi sets the content of sys.argv to ["modwsgi"] only.
Environment variables are still a possibility.

I'd be more than happy to hear any other ideas regarding this
solution. I would love to implement #2 if it is somehow possible
(environment variables, maybe?) but #1 would work but be very messy
for operators and users.

It appears that puppet is still spreading the config files for the
server
into multiple locations as well[1]. Does it inherit that logic from
devstack? Because that will need to be changed to push all of the
relevant
server config into one conf.

1.
http://logs.openstack.org/82/500182/3/check/gate-puppet-open
stack-integration-4-scenario004-tempest-ubuntu-xenial/
791523c/logs/etc/neutron/plugins/

On Sun, Sep 3, 2017 at 12:03 PM, Mohammed Naser mnaser@vexxhost.com
wrote:

On Sun, Sep 3, 2017 at 3:03 PM, Mohammed Naser mnaser@vexxhost.com
wrote:

On Sun, Sep 3, 2017 at 2:20 PM, Matthew Treinish <
mtreinish@kortar.org>
wrote:

On Sun, Sep 03, 2017 at 01:47:24PM -0400, Mohammed Naser wrote:

Hi folks,

I've attempted to enable modwsgi support in our dev environment
with
Puppet however it results in a traceback. I figured it was an
environment thing so I looked into moving the Puppet CI to test
using
mod
wsgi and it resulted in the same error.

http://logs.openstack.org/82/500182/3/check/gate-puppet-open
stack-integration-4-scenario004-tempest-ubuntu-xenial/
791523c/logs/apache/neutronwsgierror.txt.gz

Would anyone from the Neutron team be able to give input on this?
We'd love to add gating for Neutron deployed by mod_wsgi which can
help find similar issues.

Neutron never got their wsgi support working in Devstack either. The
patch
adding that: https://review.openstack.org/#/c/439191/ never passed
the
gate and
seems to have lost the attention of the author. The wsgi support in
neutron
probably doesn't work yet, and is definitely untested. IIRC, the
issue
they were
hitting was loading the config files. [1] I don't think I saw any
progress on it
after that though.

The TC goal doc [2] probably should say something about it never
landing and
missing pike.

That would make sense. The release notes also state that it does
offer the ability to run inside mod_wsgi which can be misleading to
deployers (that was the main reason I thought we can start testing
using it):

Sigh, hit send too early. Here is the link:

http://git.openstack.org/cgit/openstack/neutron/commit/?id=9
16bc96ee214078496b4b38e1c93f36f906ce840

-Matt Treinish

[1]
http://lists.openstack.org/pipermail/openstack-dev/2017-June
/117830.html
[2]
https://governance.openstack.org/tc/goals/pike/deploy-api-in
-wsgi.html#neutron



OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.op
enstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.op
enstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 4, 2017 by kevin_at_benton.pub (15,600 points)   2 3 4
0 votes

On Mon, Sep 4, 2017 at 11:07 AM, Kevin Benton kevin@benton.pub wrote:

2 would be preferable as well just because we have so many guides/distros

mentioning the different file locations. I'm not familiar with mod_wsgi
enough to know if it's possible.

I think you might have missed my message earlier by a few seconds,
looks like it's not really possible :(

Another 3rd option may be to edit the oslo config constructor call when
using that entry point to add some well-known paths for additional config
files rather than only parsing 'sys.argv[1]'. For example, we could always
try to add /etc/neutron/plugin.ini to the list which we can document that
distros/deployment tools should always create or have a symbolic link to a
plugin-specific config (e.g. ml2_conf.ini).

I believe that would be the best option, I did a bit of research and I
have the following in regards to distros:

I have mixed feelings about this. I think we should add
/etc/neutron/plugin.ini and advice Ubuntu packaging to change their
default (with a plugin.ini as a symlink). If we introduce both
ml2_conf.ini and plugin.ini, we risk creating problems if someone
is not using the ml2 plugin. Alternatively, you could load
plugin.ini and fallback to /etc/neutron/plugins/ml2/ml2_conf.ini
if there is nothing? It's a hard call but it should probably be
decided by the Neutron team in coordination with the distribution
packagers.

On Mon, Sep 4, 2017 at 7:40 AM, Mohammed Naser mnaser@vexxhost.com wrote:

On Mon, Sep 4, 2017 at 9:01 AM, Kevin Benton kevin@benton.pub wrote:

Yes, unfortunately I didn't make it back to the patch in time to adjust
devstack to dump all of the configuration into one file (instead of
/etc/neutron/neutron.conf /etc/neutron/plugins/ml2.conf etc). I did test
locally with my dev environment around the time that RPC server patch
went
in, but there may have been a regression since it went in since it's not
tested as Matt pointed out.

I've added Puppet into this because I think we would have to take a
decision regarding this. The reason behind the fact that we've always
used the two configuration files is because distributions which ship
packages actually provide 2 configuration files.

We use a configuration resource called neutron_plugin_ml2 which
configures things always in /etc/neutron/plugins/ml2/ml2_conf.ini.

In the case of Ubuntu/Debian based systems, we update
/etc/default/neutron-server to point the plugin configuration to
/etc/neutron/plugin.ini and then we ensure that the file is a
symbolic link to /etc/neutron/plugins/ml2/ml2_conf.ini.

Now, we have two options in my mind (and I am open for others):

1) Configure plugins entirely inside neutron.conf
This option would solve our issue but introduce another one. I
believe that the order in the start commands would mean that the later
configuration files (in this case, the plugin.ini) would have priority
over the neutron.conf causing an inconsistency, I don't think this
is possible. However, we can work around this by ensuring that the
plugin.ini file is empty. However, we will be introducing service
restarts for no reason with that change which can be very confusing
for the user as to why configuration is moving.

2) Figure out a way to pass 'args' via WSGI?
I'm not sure if this is entirely possible at all. But, could there be
a way that we can pass a list of configuration files in the mod_wsgi
configuration? This would make it the most transparent fix without
having to adjust all of the configuration files and bend against the
set configuration paths of the distro.

I'd be more than happy to hear any other ideas regarding this
solution. I would love to implement #2 if it is somehow possible
(environment variables, maybe?) but #1 would work but be very messy
for operators and users.

It appears that puppet is still spreading the config files for the
server
into multiple locations as well[1]. Does it inherit that logic from
devstack? Because that will need to be changed to push all of the
relevant
server config into one conf.

1.

http://logs.openstack.org/82/500182/3/check/gate-puppet-openstack-integration-4-scenario004-tempest-ubuntu-xenial/791523c/logs/etc/neutron/plugins/

On Sun, Sep 3, 2017 at 12:03 PM, Mohammed Naser mnaser@vexxhost.com
wrote:

On Sun, Sep 3, 2017 at 3:03 PM, Mohammed Naser mnaser@vexxhost.com
wrote:

On Sun, Sep 3, 2017 at 2:20 PM, Matthew Treinish
mtreinish@kortar.org
wrote:

On Sun, Sep 03, 2017 at 01:47:24PM -0400, Mohammed Naser wrote:

Hi folks,

I've attempted to enable modwsgi support in our dev environment
with
Puppet however it results in a traceback. I figured it was an
environment thing so I looked into moving the Puppet CI to test
using
mod
wsgi and it resulted in the same error.

http://logs.openstack.org/82/500182/3/check/gate-puppet-openstack-integration-4-scenario004-tempest-ubuntu-xenial/791523c/logs/apache/neutron_wsgi_error.txt.gz

Would anyone from the Neutron team be able to give input on this?
We'd love to add gating for Neutron deployed by mod_wsgi which can
help find similar issues.

Neutron never got their wsgi support working in Devstack either. The
patch
adding that: https://review.openstack.org/#/c/439191/ never passed
the
gate and
seems to have lost the attention of the author. The wsgi support in
neutron
probably doesn't work yet, and is definitely untested. IIRC, the
issue
they were
hitting was loading the config files. [1] I don't think I saw any
progress on it
after that though.

The TC goal doc [2] probably should say something about it never
landing and
missing pike.

That would make sense. The release notes also state that it does
offer the ability to run inside mod_wsgi which can be misleading to
deployers (that was the main reason I thought we can start testing
using it):

Sigh, hit send too early. Here is the link:

http://git.openstack.org/cgit/openstack/neutron/commit/?id=916bc96ee214078496b4b38e1c93f36f906ce840

-Matt Treinish

[1]

http://lists.openstack.org/pipermail/openstack-dev/2017-June/117830.html
[2]

https://governance.openstack.org/tc/goals/pike/deploy-api-in-wsgi.html#neutron


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 4, 2017 by Mohammed_Naser (3,860 points)   1 3
0 votes

Did you look into setting config files in an env var? That seemed like the
best bet given that #2 is out.

On Mon, Sep 4, 2017 at 8:19 AM, Mohammed Naser mnaser@vexxhost.com wrote:

On Mon, Sep 4, 2017 at 11:07 AM, Kevin Benton kevin@benton.pub wrote:

2 would be preferable as well just because we have so many

guides/distros
mentioning the different file locations. I'm not familiar with mod_wsgi
enough to know if it's possible.

I think you might have missed my message earlier by a few seconds,
looks like it's not really possible :(

Another 3rd option may be to edit the oslo config constructor call when
using that entry point to add some well-known paths for additional config
files rather than only parsing 'sys.argv[1]'. For example, we could
always
try to add /etc/neutron/plugin.ini to the list which we can document that
distros/deployment tools should always create or have a symbolic link to
a
plugin-specific config (e.g. ml2_conf.ini).

I believe that would be the best option, I did a bit of research and I
have the following in regards to distros:

I have mixed feelings about this. I think we should add
/etc/neutron/plugin.ini and advice Ubuntu packaging to change their
default (with a plugin.ini as a symlink). If we introduce both
ml2_conf.ini and plugin.ini, we risk creating problems if someone
is not using the ml2 plugin. Alternatively, you could load
plugin.ini and fallback to /etc/neutron/plugins/ml2/ml2_conf.ini
if there is nothing? It's a hard call but it should probably be
decided by the Neutron team in coordination with the distribution
packagers.

On Mon, Sep 4, 2017 at 7:40 AM, Mohammed Naser mnaser@vexxhost.com
wrote:

On Mon, Sep 4, 2017 at 9:01 AM, Kevin Benton kevin@benton.pub wrote:

Yes, unfortunately I didn't make it back to the patch in time to
adjust
devstack to dump all of the configuration into one file (instead of
/etc/neutron/neutron.conf /etc/neutron/plugins/ml2.conf etc). I did
test
locally with my dev environment around the time that RPC server patch
went
in, but there may have been a regression since it went in since it's
not
tested as Matt pointed out.

I've added Puppet into this because I think we would have to take a
decision regarding this. The reason behind the fact that we've always
used the two configuration files is because distributions which ship
packages actually provide 2 configuration files.

We use a configuration resource called neutron_plugin_ml2 which
configures things always in /etc/neutron/plugins/ml2/ml2_conf.ini.

In the case of Ubuntu/Debian based systems, we update
/etc/default/neutron-server to point the plugin configuration to
/etc/neutron/plugin.ini and then we ensure that the file is a
symbolic link to /etc/neutron/plugins/ml2/ml2_conf.ini.

Now, we have two options in my mind (and I am open for others):

1) Configure plugins entirely inside neutron.conf
This option would solve our issue but introduce another one. I
believe that the order in the start commands would mean that the later
configuration files (in this case, the plugin.ini) would have priority
over the neutron.conf causing an inconsistency, I don't think this
is possible. However, we can work around this by ensuring that the
plugin.ini file is empty. However, we will be introducing service
restarts for no reason with that change which can be very confusing
for the user as to why configuration is moving.

2) Figure out a way to pass 'args' via WSGI?
I'm not sure if this is entirely possible at all. But, could there be
a way that we can pass a list of configuration files in the mod_wsgi
configuration? This would make it the most transparent fix without
having to adjust all of the configuration files and bend against the
set configuration paths of the distro.

I'd be more than happy to hear any other ideas regarding this
solution. I would love to implement #2 if it is somehow possible
(environment variables, maybe?) but #1 would work but be very messy
for operators and users.

It appears that puppet is still spreading the config files for the
server
into multiple locations as well[1]. Does it inherit that logic from
devstack? Because that will need to be changed to push all of the
relevant
server config into one conf.

1.

http://logs.openstack.org/82/500182/3/check/gate-puppet-
openstack-integration-4-scenario004-tempest-ubuntu-
xenial/791523c/logs/etc/neutron/plugins/

On Sun, Sep 3, 2017 at 12:03 PM, Mohammed Naser mnaser@vexxhost.com
wrote:

On Sun, Sep 3, 2017 at 3:03 PM, Mohammed Naser mnaser@vexxhost.com
wrote:

On Sun, Sep 3, 2017 at 2:20 PM, Matthew Treinish
mtreinish@kortar.org
wrote:

On Sun, Sep 03, 2017 at 01:47:24PM -0400, Mohammed Naser wrote:

Hi folks,

I've attempted to enable modwsgi support in our dev environment
with
Puppet however it results in a traceback. I figured it was an
environment thing so I looked into moving the Puppet CI to test
using
mod
wsgi and it resulted in the same error.

http://logs.openstack.org/82/500182/3/check/gate-puppet-
openstack-integration-4-scenario004-tempest-ubuntu-
xenial/791523c/logs/apache/neutronwsgierror.txt.gz

Would anyone from the Neutron team be able to give input on this?
We'd love to add gating for Neutron deployed by mod_wsgi which
can
help find similar issues.

Neutron never got their wsgi support working in Devstack either.
The
patch
adding that: https://review.openstack.org/#/c/439191/ never
passed
the
gate and
seems to have lost the attention of the author. The wsgi support
in
neutron
probably doesn't work yet, and is definitely untested. IIRC, the
issue
they were
hitting was loading the config files. [1] I don't think I saw any
progress on it
after that though.

The TC goal doc [2] probably should say something about it never
landing and
missing pike.

That would make sense. The release notes also state that it does
offer the ability to run inside mod_wsgi which can be misleading to
deployers (that was the main reason I thought we can start testing
using it):

Sigh, hit send too early. Here is the link:

http://git.openstack.org/cgit/openstack/neutron/commit/?id=
916bc96ee214078496b4b38e1c93f36f906ce840

-Matt Treinish

[1]

http://lists.openstack.org/pipermail/openstack-dev/2017-
June/117830.html
[2]

https://governance.openstack.org/tc/goals/pike/deploy-api-
in-wsgi.html#neutron



OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:
unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:
unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 5, 2017 by kevin_at_benton.pub (15,600 points)   2 3 4
0 votes

On Mon, Sep 4, 2017 at 8:07 AM, Kevin Benton kevin@benton.pub wrote:

2 would be preferable as well just because we have so many guides/distros

mentioning the different file locations. I'm not familiar with mod_wsgi
enough to know if it's possible.

Another 3rd option may be to edit the oslo config constructor call when
using that entry point to add some well-known paths for additional config
files rather than only parsing 'sys.argv[1]'. For example, we could always
try to add /etc/neutron/plugin.ini to the list which we can document that
distros/deployment tools should always create or have a symbolic link to a
plugin-specific config (e.g. ml2_conf.ini).

For the record, oslo.config already reads /etc/neutron/neutron.conf
(and some other locations) when no arguments are passed:

https://github.com/openstack/oslo.config/blob/master/oslo_config/cfg.py#L711-L739

Also for the record, I consider not being able to load existing split
configuration files a regression. We won't be able to move to the new
operation mode unless we figure out how to load them. If mod_wsgi is
not willing to bend their argv, a envvar could be a good option.

Ihar


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 5, 2017 by Ihar_Hrachyshka (35,300 points)   3 9 16
0 votes

For the record, oslo.config already reads /etc/neutron/neutron.conf

Yeah, I think this is the only reason it even manages to get some of the
correct configuration (loading a core plugin at all).

Also for the record, I consider not being able to load existing split configuration
files a regression.

+1. That would make an upgrade nightmare. In my earlier email I didn't mean
to imply that was the only way forward. I was just curious if it still
broke when everything was in one file.

On Tue, Sep 5, 2017 at 7:01 AM, Ihar Hrachyshka ihrachys@redhat.com wrote:

On Mon, Sep 4, 2017 at 8:07 AM, Kevin Benton kevin@benton.pub wrote:

2 would be preferable as well just because we have so many

guides/distros
mentioning the different file locations. I'm not familiar with mod_wsgi
enough to know if it's possible.

Another 3rd option may be to edit the oslo config constructor call when
using that entry point to add some well-known paths for additional config
files rather than only parsing 'sys.argv[1]'. For example, we could
always
try to add /etc/neutron/plugin.ini to the list which we can document that
distros/deployment tools should always create or have a symbolic link to
a
plugin-specific config (e.g. ml2_conf.ini).

For the record, oslo.config already reads /etc/neutron/neutron.conf
(and some other locations) when no arguments are passed:

https://github.com/openstack/oslo.config/blob/master/oslo_
config/cfg.py#L711-L739

Also for the record, I consider not being able to load existing split
configuration files a regression. We won't be able to move to the new
operation mode unless we figure out how to load them. If mod_wsgi is
not willing to bend their argv, a envvar could be a good option.

Ihar


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 5, 2017 by kevin_at_benton.pub (15,600 points)   2 3 4
...