settingsLogin | Registersettings

[openstack-dev] Apache2 vs uWSGI vs ...

0 votes

In the Fuel project, we recently ran into few issues with Apache2 +
mod_wsgi as we switched Keystone to run in that environment and we started
a large battery of tests and ended up seeing these issues. Please see [1]
and [2] for examples.

Looking deep into Apache2 issues specifically around "apache2ctl graceful"
and module loading/unloading and the hooks used by modwsgi [3]. I started
wondering if Apache2 + mod
wsgi is the "right" solution and if there was
something else better that people are already using.

One data point that keeps coming up is, all the CI jobs use Apache2 +
mod_wsgi so it must be the best solution....Is it? If not, what is?

Thanks,
Dims

PS: I will leave issues with memcached + keystone for another email later :)

[1] https://bugs.launchpad.net/mos/+bug/1491576
[2] https://bugs.launchpad.net/fuel/+bug/1493372
[3] https://bugs.launchpad.net/fuel/+bug/1493372/comments/35
--
Davanum Srinivas :: https://twitter.com/dims


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Sep 17, 2015 in openstack-dev by Davanum_Srinivas (35,920 points)   2 5 9

23 Responses

0 votes

I thoughts below mention Keystone, but in reality I would apply the same
logic to any OpenStack service.

On Fri, Sep 18, 2015 at 10:32 AM, Boris Bobrov bbobrov@mirantis.com wrote:

There are 2 dimensions this discussion should happen in: web server and
application server. Now we use apache2 as web server and mod_wsgi as app
server.

This is exactly true and Keystone should be deployable in an WSGI compliant
application server. If it's not I would consider it a bug.

I don't have a specific opinion on the app server (mod_wsgi vs uwsgi) and I
don't really care.

Regarding apache2 vs nginx. I don't see any reasons for the switch.
Apache2 is
well known to deployers and sysadmins. It is very rich for modules. I
wonder
if there are customer-written modules.

Keystone doesn't use or require Apache. We recommend that it is deployed
using Apache, but there is no requirement to do that if you don't need to
use any Apache modules. For example, at least one of my devstack nodes
happily runs Keystone's manage_all.

[snip]

There are things in keystone that work under apache. They are not tested.
They
were written to work under apache because it's the simplest and the most
standard way to do. Making them work in nginx means forcing developers
write
some code. You're ready to do that?

This should only be true for optional features and currently require Apache
modules.

May be someone does not need something that apache supports and nginx not
and needs nginx features which apache does not support. Let's let our
users
decide what they want.

And the first step should be simple here - support for uwsgi.

Why uwsgi? Why not gunicorn? Cherrypy? Twisted?

uwsgi and gunicorn should both work fine, as should any WSGI application
server. CherryPy and Twister are more framework than application server, so
I would not expect them to work.

--
David
blog: http://www.traceback.org
twitter: http://twitter.com/dstanek
www: http://dstanek.com


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 18, 2015 by dstanek_at_dstanek.c (2,440 points)   1 2
0 votes

Folks

I just suggested to untie keystone from wsgi and implement uwsgi support.
And then let the user decide what he or she wants.

There is a plenty of auth modules for nginx also.

Nginx us much better as a proxy server and you know it.

Regarding mod wsgi and apache we already saw that it cannot handle simple
restart. I think this is not in any way acceptable from operations point if
view.
18 сент. 2015 г. 18:59 пользователь "Fox, Kevin M" Kevin.Fox@pnnl.gov
написал:

Part of the reason to use Apache though is the diverse set of
authentication mechanisms it supports. Operators have the desire to plugin
Keystone into their existing authentication systems and Apache tends to be
easier to do that then others.

Thanks,
Kevin


From: Jim Rollenhagen [jim@jimrollenhagen.com]
Sent: Thursday, September 17, 2015 7:04 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] Apache2 vs uWSGI vs ...

On Thu, Sep 17, 2015 at 06:48:50PM -0400, Davanum Srinivas wrote:

In the fuel project, we recently ran into a couple of issues with
Apache2 +
mod_wsgi as we switched Keystone to run . Please see [1] and [2].

Looking deep into Apache2 issues specifically around "apache2ctl
graceful"
and module loading/unloading and the hooks used by modwsgi [3]. I
started
wondering if Apache2 + mod
wsgi is the "right" solution and if there was
something else better that people are already using.

One data point that keeps coming up is, all the CI jobs use Apache2 +
mod_wsgi so it must be the best solution....Is it? If not, what is?

Disclaimer: it's been a while since I've cared about performance with a
web server in front of a Python app.

IIRC, mod_wsgi was abandoned for a while, but I think it's being worked
on again. In general, I seem to remember it being thought of as a bit
old and crusty, but mostly working.

At a previous job, we switched from Apache2 + mod_wsgi to nginx + uwsgi[0]
and saw a significant performance increase. This was a Django app. uwsgi
is fairly straightforward to operate and comes loaded with a myriad of
options[1] to help folks make the most of it. I've played with Ironic
behind uwsgi and it seemed to work fine, though I haven't done any sort
of load testing. I'd encourage folks to give it a shot. :)

Of course, uwsgi can also be ran behind Apache2, if you'd prefer.

gunicorn[2] is another good option that may be worth investigating; I
personally don't have any experience with it, but I seem to remember
hearing it has good eventlet support.

// jim

[0] https://uwsgi-docs.readthedocs.org/en/latest/
[1] https://uwsgi-docs.readthedocs.org/en/latest/Options.html
[2] http://gunicorn.org/


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 18, 2015 by Vladimir_Kuklin (7,320 points)   1 3 4
0 votes

I am with Adam, I kinda doubt Apache cause performance issues for Keystone, especially since all we have are just simple REST APIs. For other services with specific needs, like large file streaming, there may be some arguments to pick one over the other. We haven’t had the need to use Apache for dynamic routing or proxying at deployment. I am sure there are better tools out there that can do the job.

Making Keystone web service independent is a fine goal. However, since Keystone is moving away from being an identity provider, federation and external auth will play a major part going forward. Some of them are depended on the Apache at the moment. We may have to refactor Keystone to isolate the authentication service from the rest, then figure out how to gate it with Apache. I don’t think that’s trivial work though.

At this point, I don’t know what we are really gaining by ripping out Apache, comparing to amount of work to make that happen.

Guang

From: Vladimir Kuklin [mailto:vkuklin@mirantis.com]
Sent: Friday, September 18, 2015 9:09 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: [openstack-dev] Apache2 vs uWSGI vs ...

Folks

I just suggested to untie keystone from wsgi and implement uwsgi support. And then let the user decide what he or she wants.

There is a plenty of auth modules for nginx also.

Nginx us much better as a proxy server and you know it.

Regarding mod wsgi and apache we already saw that it cannot handle simple restart. I think this is not in any way acceptable from operations point if view.
18 сент. 2015 г. 18:59 пользователь "Fox, Kevin M" Kevin.Fox@pnnl.gov написал:
Part of the reason to use Apache though is the diverse set of authentication mechanisms it supports. Operators have the desire to plugin Keystone into their existing authentication systems and Apache tends to be easier to do that then others.

Thanks,
Kevin


From: Jim Rollenhagen [jim@jimrollenhagen.com]
Sent: Thursday, September 17, 2015 7:04 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] Apache2 vs uWSGI vs ...

On Thu, Sep 17, 2015 at 06:48:50PM -0400, Davanum Srinivas wrote:
In the fuel project, we recently ran into a couple of issues with Apache2 +
mod_wsgi as we switched Keystone to run . Please see [1] and [2].

Looking deep into Apache2 issues specifically around "apache2ctl graceful"
and module loading/unloading and the hooks used by modwsgi [3]. I started
wondering if Apache2 + mod
wsgi is the "right" solution and if there was
something else better that people are already using.

One data point that keeps coming up is, all the CI jobs use Apache2 +
mod_wsgi so it must be the best solution....Is it? If not, what is?

Disclaimer: it's been a while since I've cared about performance with a
web server in front of a Python app.

IIRC, mod_wsgi was abandoned for a while, but I think it's being worked
on again. In general, I seem to remember it being thought of as a bit
old and crusty, but mostly working.

At a previous job, we switched from Apache2 + mod_wsgi to nginx + uwsgi[0]
and saw a significant performance increase. This was a Django app. uwsgi
is fairly straightforward to operate and comes loaded with a myriad of
options[1] to help folks make the most of it. I've played with Ironic
behind uwsgi and it seemed to work fine, though I haven't done any sort
of load testing. I'd encourage folks to give it a shot. :)

Of course, uwsgi can also be ran behind Apache2, if you'd prefer.

gunicorn[2] is another good option that may be worth investigating; I
personally don't have any experience with it, but I seem to remember
hearing it has good eventlet support.

// jim

[0] https://uwsgi-docs.readthedocs.org/en/latest/
[1] https://uwsgi-docs.readthedocs.org/en/latest/Options.html
[2] http://gunicorn.org/


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 18, 2015 by guang.yee_at_hpe.com (260 points)  
0 votes

The conversations around alternative wsgi containers (uwsgi, gunicorn, etc) still would be tied to apache or nginx. While it is possible to deploy without the webservers in some cases, jt would not be recommended nor do I see that being tested in gate. We want to leverage the auth support of the webserver. While uWSGI has interesting features, it wouldnt really replace nginx or apache. It is just an alternative deployment strategy that makes managing the keystone (or any OpenStack service) processes independent of the web server.

I do not expect any of these choices to have a performance impact. It will ease some operational concerns. There is no reason uwsgi or gunicorn wont work today (I have run keystone with both in a devstack locally). The documentation and communicating the reasons to pick one or the other is all that is needed. From a gate perspective configuration and restarts of the web server could be made easier in devstack with uWSGI. It would not prevent use of mod_wsgi.

--Morgan

Sent via mobile

On Sep 18, 2015, at 09:53, Yee, Guang guang.yee@hpe.com wrote:

I am with Adam, I kinda doubt Apache cause performance issues for Keystone, especially since all we have are just simple REST APIs. For other services with specific needs, like large file streaming, there may be some arguments to pick one over the other. We haven’t had the need to use Apache for dynamic routing or proxying at deployment. I am sure there are better tools out there that can do the job.

Making Keystone web service independent is a fine goal. However, since Keystone is moving away from being an identity provider, federation and external auth will play a major part going forward. Some of them are depended on the Apache at the moment. We may have to refactor Keystone to isolate the authentication service from the rest, then figure out how to gate it with Apache. I don’t think that’s trivial work though.

At this point, I don’t know what we are really gaining by ripping out Apache, comparing to amount of work to make that happen.


Guang


From: Vladimir Kuklin [mailto:vkuklin@mirantis.com]
Sent: Friday, September 18, 2015 9:09 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: [openstack-dev] Apache2 vs uWSGI vs ...

Folks

I just suggested to untie keystone from wsgi and implement uwsgi support. And then let the user decide what he or she wants.

There is a plenty of auth modules for nginx also.

Nginx us much better as a proxy server and you know it.

Regarding mod wsgi and apache we already saw that it cannot handle simple restart. I think this is not in any way acceptable from operations point if view.

18 сент. 2015 г. 18:59 пользователь "Fox, Kevin M" Kevin.Fox@pnnl.gov написал:
Part of the reason to use Apache though is the diverse set of authentication mechanisms it supports. Operators have the desire to plugin Keystone into their existing authentication systems and Apache tends to be easier to do that then others.

Thanks,
Kevin


From: Jim Rollenhagen [jim@jimrollenhagen.com]
Sent: Thursday, September 17, 2015 7:04 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] Apache2 vs uWSGI vs ...

On Thu, Sep 17, 2015 at 06:48:50PM -0400, Davanum Srinivas wrote:

In the fuel project, we recently ran into a couple of issues with Apache2 +
mod_wsgi as we switched Keystone to run . Please see [1] and [2].

Looking deep into Apache2 issues specifically around "apache2ctl graceful"
and module loading/unloading and the hooks used by modwsgi [3]. I started
wondering if Apache2 + mod
wsgi is the "right" solution and if there was
something else better that people are already using.

One data point that keeps coming up is, all the CI jobs use Apache2 +
mod_wsgi so it must be the best solution....Is it? If not, what is?

Disclaimer: it's been a while since I've cared about performance with a
web server in front of a Python app.

IIRC, mod_wsgi was abandoned for a while, but I think it's being worked
on again. In general, I seem to remember it being thought of as a bit
old and crusty, but mostly working.

At a previous job, we switched from Apache2 + mod_wsgi to nginx + uwsgi[0]
and saw a significant performance increase. This was a Django app. uwsgi
is fairly straightforward to operate and comes loaded with a myriad of
options[1] to help folks make the most of it. I've played with Ironic
behind uwsgi and it seemed to work fine, though I haven't done any sort
of load testing. I'd encourage folks to give it a shot. :)

Of course, uwsgi can also be ran behind Apache2, if you'd prefer.

gunicorn[2] is another good option that may be worth investigating; I
personally don't have any experience with it, but I seem to remember
hearing it has good eventlet support.

// jim

[0] https://uwsgi-docs.readthedocs.org/en/latest/
[1] https://uwsgi-docs.readthedocs.org/en/latest/Options.html
[2] http://gunicorn.org/


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 18, 2015 by Morgan_Fainberg (17,320 points)   2 6 9
0 votes

On Fri, Sep 18, 2015 at 11:09 AM, Vladimir Kuklin vkuklin@mirantis.com
wrote:

I just suggested to untie keystone from wsgi and implement uwsgi support.
And then let the user decide what he or she wants.

Keystone is not tied to Apache or modwsgi, if that's what you mean. We
provide a sample configuration for Apache + mod
wsgi because deployers tend
to be more familiar with Apache than other web servers, and there happens
to be a mature ecosystem of auth modules that keystone can utilize. It's
working documentation, just like devstack itself.

Use whatever you want to deploy keystone. Choose the thing that you're
familiar with, can support effectively, run reliably, and has sufficient
performance.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 18, 2015 by Dolph_Mathews (9,560 points)   1 2 3
0 votes

Hi,

Morgan gave the perfect case why operators want to use uWSGI. Let's imagine
a future when all openstack services will work as modwsgi processes under
apache. It's like to put all eggs in one basket. If you need to reconfigure
one service on controller it may affect another service. For instance,
sometimes operators need to increase number of Threads/Processes for wsgi
or add new virtual host to apache. That will require graceful or cold
restart of apache. It affects other services. Another case, internal
problems in mod
wsgi where it may lead to apache crash affecting all
services.

uWSGI/gunicorn model is safer as in this case apache is reverse_proxy only.
This model gives flexibility for operators. They may use apache/nginx as
proxy or load balancer. Stop or crash of one service won't lead to downtime
of other services. The complexity of OpenStack management will be easier
and friendly.

--
Best regards,
Sergii Golovatiuk,
Skype #golserge
IRC #holser

On Fri, Sep 18, 2015 at 3:44 PM, Morgan Fainberg morgan.fainberg@gmail.com
wrote:

There is and has been desire to support uWSGI and other alternatives to
modwsgi. There are a variety of operational reasons to consider uWSGI
and/or gunicorn behind apache most notably to facilitate easier management
of the processes independently of the webserver itself. With mod
wsgi the
processes are directly tied to the apache server where as with uWSGI and
gunicorn you can manage the various services independently and/or with
differing VENVs more easily.

There are potential other concerns that must be weighed when considering
which method of deployment to use. I hope we have clear documentation
within the next cycle (and possible choices for the gate) for utilizing
uWSGI and/or gunicorn.

--Morgan

Sent via mobile

On Sep 18, 2015, at 06:12, Adam Young ayoung@redhat.com wrote:

On 09/17/2015 10:04 PM, Jim Rollenhagen wrote:

On Thu, Sep 17, 2015 at 06:48:50PM -0400, Davanum Srinivas wrote:

In the fuel project, we recently ran into a couple of issues with Apache2 +
mod_wsgi as we switched Keystone to run . Please see [1] and [2].

Looking deep into Apache2 issues specifically around "apache2ctl graceful"
and module loading/unloading and the hooks used by modwsgi [3]. I started
wondering if Apache2 + mod
wsgi is the "right" solution and if there was
something else better that people are already using.

One data point that keeps coming up is, all the CI jobs use Apache2 +
mod_wsgi so it must be the best solution....Is it? If not, what is?

Disclaimer: it's been a while since I've cared about performance with a
web server in front of a Python app.

IIRC, mod_wsgi was abandoned for a while, but I think it's being worked
on again. In general, I seem to remember it being thought of as a bit
old and crusty, but mostly working.

I am not aware of that. It has been the workhorse of the Python/wsgi
world for a while, and we use it heavily.

At a previous job, we switched from Apache2 + mod_wsgi to nginx + uwsgi[0]
and saw a significant performance increase. This was a Django app. uwsgi
is fairly straightforward to operate and comes loaded with a myriad of
options[1] to help folks make the most of it. I've played with Ironic
behind uwsgi and it seemed to work fine, though I haven't done any sort
of load testing. I'd encourage folks to give it a shot. :)

Again, switching web servers is as likely to introduce as to solve
problems. If there are performance issues:

  1. Idenitfy what causes them
  2. Change configuration settings to deal with them
  3. Fix upstream bugs in the underlying system.

Keystone is not about performance. Keystone is about security. The cloud
is designed to scale horizontally first. Before advocating switching to a
difference web server, make sure it supports the technologies required.

  1. TLS at the latest level
  2. Kerberos/GSSAPI/SPNEGO
  3. X509 Client cert validation
  4. SAML

OpenID connect would be a good one to add to the list; Its been requested
for a while.

If Keystone is having performance issues, it is most likely at the
database layer, not the web server.

"Programmers waste enormous amounts of time thinking about, or worrying
about, the speed of noncritical parts of their programs, and these attempts
at efficiency actually have a strong negative impact when debugging and
maintenance are considered. We should forget about small efficiencies,
say about 97% of the time: premature optimization is the root of all
evil.
Yet we should not pass up our opportunities in that critical
3%." --Donald Knuth

Of course, uwsgi can also be ran behind Apache2, if you'd prefer.

gunicorn[2] is another good option that may be worth investigating; I
personally don't have any experience with it, but I seem to remember
hearing it has good eventlet support.

// jim

[0] https://uwsgi-docs.readthedocs.org/en/latest/
[1] https://uwsgi-docs.readthedocs.org/en/latest/Options.html
[2] http://gunicorn.org/


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribehttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 25, 2015 by Sergii_Golovatiuk (6,080 points)   1 2 3
0 votes

Alexandr,

oauth, shibboleth & openid support are very keystone specific features.
Many other openstack projects don't need these modules at all but they may
require faster HTTP server (lighthttp/nginx).

For all projects we may use "HTTP server -> uwsgi" model and leave apache
for keystone as " HTTP server -> apache -> uwsgi/mod_wsgi". However, I
would like to think about whole Openstack ecosystem in general. In that
case we'll minimize the number of programs operator should have knowledge.

--
Best regards,
Sergii Golovatiuk,
Skype #golserge
IRC #holser

On Fri, Sep 18, 2015 at 4:28 PM, Alexander Makarov amakarov@mirantis.com
wrote:

Please consider that we use some apache mods - does
nginx/uwsgi/gunicorn have oauth, shibboleth & openid support?

On Fri, Sep 18, 2015 at 4:54 PM, Vladimir Kuklin vkuklin@mirantis.com
wrote:

Folks

I think we do not need to switch to nginx-only or consider any kind of
war
between nginx and apache adherents. Everyone should be able to use
web-server he or she needs without being pinned to the unwanted one. It
is
like Postgres vs MySQL war. Why not support both?

May be someone does not need something that apache supports and nginx not
and needs nginx features which apache does not support. Let's let our
users
decide what they want.

And the first step should be simple here - support for uwsgi. It will
allow
for usage of any web-server that can work with uwsgi. It will allow also
us
to check for the support of all apache-like bindings like SPNEGO or
whatever
and provide our users with enough info on making decisions. I did not
personally test nginx modules for SAML and SPNEGO, but I am pretty
confident
about TLS/SSL parts of nginx.

Moreover, nginx will allow you to do things you cannot do with apache,
e.g.
do smart load balancing, which may be crucial for high-loaded
installations.

On Fri, Sep 18, 2015 at 4:12 PM, Adam Young ayoung@redhat.com wrote:

On 09/17/2015 10:04 PM, Jim Rollenhagen wrote:

On Thu, Sep 17, 2015 at 06:48:50PM -0400, Davanum Srinivas wrote:

In the fuel project, we recently ran into a couple of issues with
Apache2
+
mod_wsgi as we switched Keystone to run . Please see [1] and [2].

Looking deep into Apache2 issues specifically around "apache2ctl
graceful"
and module loading/unloading and the hooks used by modwsgi [3]. I
started
wondering if Apache2 + mod
wsgi is the "right" solution and if there was
something else better that people are already using.

One data point that keeps coming up is, all the CI jobs use Apache2 +
mod_wsgi so it must be the best solution....Is it? If not, what is?

Disclaimer: it's been a while since I've cared about performance with a
web server in front of a Python app.

IIRC, mod_wsgi was abandoned for a while, but I think it's being worked
on again. In general, I seem to remember it being thought of as a bit
old and crusty, but mostly working.

I am not aware of that. It has been the workhorse of the Python/wsgi
world for a while, and we use it heavily.

At a previous job, we switched from Apache2 + mod_wsgi to nginx +
uwsgi[0]
and saw a significant performance increase. This was a Django app. uwsgi
is fairly straightforward to operate and comes loaded with a myriad of
options[1] to help folks make the most of it. I've played with Ironic
behind uwsgi and it seemed to work fine, though I haven't done any sort
of load testing. I'd encourage folks to give it a shot. :)

Again, switching web servers is as likely to introduce as to solve
problems. If there are performance issues:

  1. Idenitfy what causes them
  2. Change configuration settings to deal with them
  3. Fix upstream bugs in the underlying system.

Keystone is not about performance. Keystone is about security. The
cloud
is designed to scale horizontally first. Before advocating switching
to a
difference web server, make sure it supports the technologies required.

  1. TLS at the latest level
  2. Kerberos/GSSAPI/SPNEGO
  3. X509 Client cert validation
  4. SAML

OpenID connect would be a good one to add to the list; Its been
requested
for a while.

If Keystone is having performance issues, it is most likely at the
database layer, not the web server.

"Programmers waste enormous amounts of time thinking about, or worrying
about, the speed of noncritical parts of their programs, and these
attempts
at efficiency actually have a strong negative impact when debugging and
maintenance are considered. We should forget about small efficiencies,
say
about 97% of the time: premature optimization is the root of all evil.
Yet
we should not pass up our opportunities in that critical 3%." --Donald
Knuth

Of course, uwsgi can also be ran behind Apache2, if you'd prefer.

gunicorn[2] is another good option that may be worth investigating; I
personally don't have any experience with it, but I seem to remember
hearing it has good eventlet support.

// jim

[0] https://uwsgi-docs.readthedocs.org/en/latest/
[1] https://uwsgi-docs.readthedocs.org/en/latest/Options.html
[2] http://gunicorn.org/


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Yours Faithfully,
Vladimir Kuklin,
Fuel Library Tech Lead,
Mirantis, Inc.
+7 (495) 640-49-04
+7 (926) 702-39-68
Skype kuklinvv
35bk3, Vorontsovskaya Str.
Moscow, Russia,
www.mirantis.com
www.mirantis.ru
vkuklin@mirantis.com


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Kind Regards,
Alexander Makarov,
Senior Software Developer,

Mirantis, Inc.
35b/3, Vorontsovskaya St., 109147, Moscow, Russia

Tel.: +7 (495) 640-49-04
Tel.: +7 (926) 204-50-60

Skype: MAKAPOB.AJIEKCAHDP


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 25, 2015 by Sergii_Golovatiuk (6,080 points)   1 2 3
0 votes

Are we discussing mod_wsgi and Keystone or OpenStack as a general?
If Keystone specific use case, then probably Apache provides broadest
choice of tested external authenticators.
I'm not against uwsgi at all, but to be honest expectation that nginx could
substitute Apache in terms of authentication providers is simply
unrealistic.

A.

On Fri, Sep 25, 2015 at 1:24 PM, Sergii Golovatiuk <sgolovatiuk@mirantis.com
wrote:

Alexandr,

oauth, shibboleth & openid support are very keystone specific features.
Many other openstack projects don't need these modules at all but they may
require faster HTTP server (lighthttp/nginx).

For all projects we may use "HTTP server -> uwsgi" model and leave apache
for keystone as " HTTP server -> apache -> uwsgi/mod_wsgi". However, I
would like to think about whole Openstack ecosystem in general. In that
case we'll minimize the number of programs operator should have knowledge.

--
Best regards,
Sergii Golovatiuk,
Skype #golserge
IRC #holser

On Fri, Sep 18, 2015 at 4:28 PM, Alexander Makarov amakarov@mirantis.com
wrote:

Please consider that we use some apache mods - does
nginx/uwsgi/gunicorn have oauth, shibboleth & openid support?

On Fri, Sep 18, 2015 at 4:54 PM, Vladimir Kuklin vkuklin@mirantis.com
wrote:

Folks

I think we do not need to switch to nginx-only or consider any kind of
war
between nginx and apache adherents. Everyone should be able to use
web-server he or she needs without being pinned to the unwanted one. It
is
like Postgres vs MySQL war. Why not support both?

May be someone does not need something that apache supports and nginx
not
and needs nginx features which apache does not support. Let's let our
users
decide what they want.

And the first step should be simple here - support for uwsgi. It will
allow
for usage of any web-server that can work with uwsgi. It will allow
also us
to check for the support of all apache-like bindings like SPNEGO or
whatever
and provide our users with enough info on making decisions. I did not
personally test nginx modules for SAML and SPNEGO, but I am pretty
confident
about TLS/SSL parts of nginx.

Moreover, nginx will allow you to do things you cannot do with apache,
e.g.
do smart load balancing, which may be crucial for high-loaded
installations.

On Fri, Sep 18, 2015 at 4:12 PM, Adam Young ayoung@redhat.com wrote:

On 09/17/2015 10:04 PM, Jim Rollenhagen wrote:

On Thu, Sep 17, 2015 at 06:48:50PM -0400, Davanum Srinivas wrote:

In the fuel project, we recently ran into a couple of issues with
Apache2
+
mod_wsgi as we switched Keystone to run . Please see [1] and [2].

Looking deep into Apache2 issues specifically around "apache2ctl
graceful"
and module loading/unloading and the hooks used by modwsgi [3]. I
started
wondering if Apache2 + mod
wsgi is the "right" solution and if there
was
something else better that people are already using.

One data point that keeps coming up is, all the CI jobs use Apache2 +
mod_wsgi so it must be the best solution....Is it? If not, what is?

Disclaimer: it's been a while since I've cared about performance with a
web server in front of a Python app.

IIRC, mod_wsgi was abandoned for a while, but I think it's being worked
on again. In general, I seem to remember it being thought of as a bit
old and crusty, but mostly working.

I am not aware of that. It has been the workhorse of the Python/wsgi
world for a while, and we use it heavily.

At a previous job, we switched from Apache2 + mod_wsgi to nginx +
uwsgi[0]
and saw a significant performance increase. This was a Django app.
uwsgi
is fairly straightforward to operate and comes loaded with a myriad of
options[1] to help folks make the most of it. I've played with Ironic
behind uwsgi and it seemed to work fine, though I haven't done any sort
of load testing. I'd encourage folks to give it a shot. :)

Again, switching web servers is as likely to introduce as to solve
problems. If there are performance issues:

  1. Idenitfy what causes them
  2. Change configuration settings to deal with them
  3. Fix upstream bugs in the underlying system.

Keystone is not about performance. Keystone is about security. The
cloud
is designed to scale horizontally first. Before advocating switching
to a
difference web server, make sure it supports the technologies required.

  1. TLS at the latest level
  2. Kerberos/GSSAPI/SPNEGO
  3. X509 Client cert validation
  4. SAML

OpenID connect would be a good one to add to the list; Its been
requested
for a while.

If Keystone is having performance issues, it is most likely at the
database layer, not the web server.

"Programmers waste enormous amounts of time thinking about, or worrying
about, the speed of noncritical parts of their programs, and these
attempts
at efficiency actually have a strong negative impact when debugging and
maintenance are considered. We should forget about small efficiencies,
say
about 97% of the time: premature optimization is the root of all evil.
Yet
we should not pass up our opportunities in that critical 3%."
--Donald
Knuth

Of course, uwsgi can also be ran behind Apache2, if you'd prefer.

gunicorn[2] is another good option that may be worth investigating; I
personally don't have any experience with it, but I seem to remember
hearing it has good eventlet support.

// jim

[0] https://uwsgi-docs.readthedocs.org/en/latest/
[1] https://uwsgi-docs.readthedocs.org/en/latest/Options.html
[2] http://gunicorn.org/


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Yours Faithfully,
Vladimir Kuklin,
Fuel Library Tech Lead,
Mirantis, Inc.
+7 (495) 640-49-04
+7 (926) 702-39-68
Skype kuklinvv
35bk3, Vorontsovskaya Str.
Moscow, Russia,
www.mirantis.com
www.mirantis.ru
vkuklin@mirantis.com


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Kind Regards,
Alexander Makarov,
Senior Software Developer,

Mirantis, Inc.
35b/3, Vorontsovskaya St., 109147, Moscow, Russia

Tel.: +7 (495) 640-49-04
Tel.: +7 (926) 204-50-60

Skype: MAKAPOB.AJIEKCAHDP


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Adam Heczko
Security Engineer @ Mirantis Inc.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 25, 2015 by Adam_Heczko (1,860 points)   1
0 votes

On Fri, Sep 25, 2015 at 8:25 AM Adam Heczko aheczko@mirantis.com wrote:

Are we discussing mod_wsgi and Keystone or OpenStack as a general?
If Keystone specific use case, then probably Apache provides broadest
choice of tested external authenticators.
I'm not against uwsgi at all, but to be honest expectation that nginx
could substitute Apache in terms of authentication providers is simply
unrealistic.

uwsgi isn't a replacement for Apache. It's a replacement for mod_wsgi. It
just so happens that it does let you use user web servers if that's what
your usecase dictates.

As a Keystone developer I don't want to tell deployers that they have to
use Apache. It should be their choice. Since Apache is the most common web
server in our community I think we should continue to provide example
configurations and guidance for it.

-- David


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 25, 2015 by dstanek_at_dstanek.c (2,440 points)   1 2
0 votes

David,

I am thinking how all OpenStack components may coexist as mod_wsgi
processes under apache. In Fuel we stepped into problem with deployment
using graceful restart [1] thus this thread was raised to have a good WSGI
alternatives.

[1] https://github.com/GrahamDumpleton/mod_wsgi/pull/95

--
Best regards,
Sergii Golovatiuk,
Skype #golserge
IRC #holser


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 25, 2015 by Sergii_Golovatiuk (6,080 points)   1 2 3
...