settingsLogin | Registersettings

[openstack-dev] Apache2 vs uWSGI vs ...

0 votes

In the Fuel project, we recently ran into few issues with Apache2 +
mod_wsgi as we switched Keystone to run in that environment and we started
a large battery of tests and ended up seeing these issues. Please see [1]
and [2] for examples.

Looking deep into Apache2 issues specifically around "apache2ctl graceful"
and module loading/unloading and the hooks used by modwsgi [3]. I started
wondering if Apache2 + mod
wsgi is the "right" solution and if there was
something else better that people are already using.

One data point that keeps coming up is, all the CI jobs use Apache2 +
mod_wsgi so it must be the best solution....Is it? If not, what is?

Thanks,
Dims

PS: I will leave issues with memcached + keystone for another email later :)

[1] https://bugs.launchpad.net/mos/+bug/1491576
[2] https://bugs.launchpad.net/fuel/+bug/1493372
[3] https://bugs.launchpad.net/fuel/+bug/1493372/comments/35
--
Davanum Srinivas :: https://twitter.com/dims


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Sep 17, 2015 in openstack-dev by Davanum_Srinivas (35,920 points)   2 5 9

23 Responses

0 votes

OK, sorry I mixed up nginx and uwsgi :)

A.

On Fri, Sep 25, 2015 at 2:54 PM, David Stanek dstanek@dstanek.com wrote:

On Fri, Sep 25, 2015 at 8:25 AM Adam Heczko aheczko@mirantis.com wrote:

Are we discussing mod_wsgi and Keystone or OpenStack as a general?
If Keystone specific use case, then probably Apache provides broadest
choice of tested external authenticators.
I'm not against uwsgi at all, but to be honest expectation that nginx
could substitute Apache in terms of authentication providers is simply
unrealistic.

uwsgi isn't a replacement for Apache. It's a replacement for mod_wsgi. It
just so happens that it does let you use user web servers if that's what
your usecase dictates.

As a Keystone developer I don't want to tell deployers that they have to
use Apache. It should be their choice. Since Apache is the most common web
server in our community I think we should continue to provide example
configurations and guidance for it.

-- David


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Adam Heczko
Security Engineer @ Mirantis Inc.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 25, 2015 by Adam_Heczko (1,860 points)   1
0 votes

On 09/25/2015 07:09 AM, Sergii Golovatiuk wrote:
Hi,

Morgan gave the perfect case why operators want to use uWSGI. Let's
imagine a future when all openstack services will work as modwsgi
processes under apache. It's like to put all eggs in one basket. If
you need to reconfigure one service on controller it may affect
another service. For instance, sometimes operators need to increase
number of Threads/Processes for wsgi or add new virtual host to
apache. That will require graceful or cold restart of apache. It
affects other services. Another case, internal problems in mod
wsgi
where it may lead to apache crash affecting all services.

uWSGI/gunicorn model is safer as in this case apache is reverse_proxy
only. This model gives flexibility for operators. They may use
apache/nginx as proxy or load balancer. Stop or crash of one service
won't lead to downtime of other services. The complexity of OpenStack
management will be easier and friendly.

There are some fallacies here:

  1. OpenStack services should all be on the same machine.
  2. OpenStack web services should run on ports other than 443.

I think both of these are ideas who's time have come and gone.

If you have a single machine, run them out of separate containers. That
allows different services to work with different versions of the
libraries. It lets you mix a newer Keystone with older everything else.

If everything is on port 443, you need a single web server at the front
end to multiplex it; uWSGI or any other one does not obviate that.

There are no good ports left in /etc/services; stop trying to reserve
new ones for the web. If you need to run on a web service, you need to
be able to get through firewalls. You need to run on standard ports.
Run on 443.

Keystone again is a great example of this: it has two ports: 5000 and 35357.

port 5000 in /etc/services is

commplex-main 5000/tcp

and port 35357 is smack dab in the middle of the ephemeral range.

Again, so long as the web server supports the cryptographic secure
mechanisms, I don't care what one you chose. But The idea of use going
to Keystone and getting a bearer token as the basis for security is
immature; we should be doing the following on every call:

  1. TLS
  2. Cryptographic authentication.

They can be together or split up.

So, lets get everything running inside Apache, and, at the same time,
push our other favorite web servers to support the necessary pieces to
make OpenStack and the Web secure.

--
Best regards,
Sergii Golovatiuk,
Skype #golserge
IRC #holser

On Fri, Sep 18, 2015 at 3:44 PM, Morgan Fainberg
<morgan.fainberg@gmail.com morgan.fainberg@gmail.com> wrote:

There is and has been desire to support uWSGI and other
alternatives to mod_wsgi. There are a variety of operational
reasons to consider uWSGI and/or gunicorn behind apache most
notably to facilitate easier management of the processes
independently of the webserver itself. With mod_wsgi the processes
are directly tied to the apache server where as with uWSGI and
gunicorn you can manage the various services independently and/or
with differing VENVs more easily.

There are potential other concerns that must be weighed when
considering which method of deployment to use. I hope we have
clear documentation within the next cycle (and possible choices
for the gate) for utilizing uWSGI and/or gunicorn.

--Morgan

Sent via mobile

On Sep 18, 2015, at 06:12, Adam Young <ayoung@redhat.com
<mailto:ayoung@redhat.com>> wrote:
On 09/17/2015 10:04 PM, Jim Rollenhagen wrote:
On Thu, Sep 17, 2015 at 06:48:50PM -0400, Davanum Srinivas wrote:
In the fuel project, we recently ran into a couple of issues with Apache2 +
mod_wsgi as we switched Keystone to run . Please see [1] and [2].

Looking deep into Apache2 issues specifically around "apache2ctl graceful"
and module loading/unloading and the hooks used by mod_wsgi [3]. I started
wondering if Apache2 + mod_wsgi is the "right" solution and if there was
something else better that people are already using.

One data point that keeps coming up is, all the CI jobs use Apache2 +
mod_wsgi so it must be the best solution....Is it? If not, what is?
Disclaimer: it's been a while since I've cared about performance with a
web server in front of a Python app.
IIRC, mod_wsgi was abandoned for a while, but I think it's being worked
on again. In general, I seem to remember it being thought of as a bit
old and crusty, but mostly working.
I am not aware of that.  It has been the workhorse of the
Python/wsgi world for a while, and we use it heavily.
At a previous job, we switched from Apache2 + mod_wsgi to nginx + uwsgi[0]
and saw a significant performance increase. This was a Django app. uwsgi
is fairly straightforward to operate and comes loaded with a myriad of
options[1] to help folks make the most of it. I've played with Ironic
behind uwsgi and it seemed to work fine, though I haven't done any sort
of load testing. I'd encourage folks to give it a shot. :)
Again, switching web servers is as likely to introduce as to
solve problems.  If there are performance issues:

1.  Idenitfy what causes them
2.  Change configuration settings to deal with them
3.  Fix upstream bugs in the underlying system.


Keystone is not about performance.  Keystone is about security. 
The cloud is designed to scale horizontally first.  Before
advocating switching to a difference web server, make sure it
supports the technologies required.


1. TLS at the latest level
2. Kerberos/GSSAPI/SPNEGO
3. X509 Client cert validation
4. SAML

OpenID connect would be a good one to add to the list;  Its been
requested for a while.

If Keystone is having performance issues, it is most likely at
the database layer, not the web server.



"Programmers waste enormous amounts of time thinking about, or
worrying about, the speed of noncritical parts of their programs,
and these attempts at efficiency actually have a strong negative
impact when debugging and maintenance are considered. We /should/
forget about small efficiencies, say about 97% of the time:
*premature optimization is the root of all evil.* Yet we should
not pass up our opportunities in that critical 3%."   --Donald Knuth
Of course, uwsgi can also be ran behind Apache2, if you'd prefer.

gunicorn[2] is another good option that may be worth investigating; I
personally don't have any experience with it, but I seem to remember
hearing it has good eventlet support.

// jim

[0]https://uwsgi-docs.readthedocs.org/en/latest/
[1]https://uwsgi-docs.readthedocs.org/en/latest/Options.html
[2]http://gunicorn.org/

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
<mailto:OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org
<mailto:OpenStack-dev-request@lists.openstack.org>?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 25, 2015 by Adam_Young (19,940 points)   2 7 12
0 votes

There is no reason why the wsgi app container matters. This is simply a "we should document use if uwsgi and/or gunicorn as an alternative to mod_wsgi". If one solution is better for the gate it will be used there and each deployment will make the determination of what they want to use. Adam's point remains regardless of what wsgi solution is used.

On Sep 25, 2015, at 09:23, Adam Young ayoung@redhat.com wrote:

On 09/25/2015 07:09 AM, Sergii Golovatiuk wrote:
Hi,

Morgan gave the perfect case why operators want to use uWSGI. Let's imagine a future when all openstack services will work as modwsgi processes under apache. It's like to put all eggs in one basket. If you need to reconfigure one service on controller it may affect another service. For instance, sometimes operators need to increase number of Threads/Processes for wsgi or add new virtual host to apache. That will require graceful or cold restart of apache. It affects other services. Another case, internal problems in modwsgi where it may lead to apache crash affecting all services.

uWSGI/gunicorn model is safer as in this case apache is reverse_proxy only. This model gives flexibility for operators. They may use apache/nginx as proxy or load balancer. Stop or crash of one service won't lead to downtime of other services. The complexity of OpenStack management will be easier and friendly.

There are some fallacies here:

  1. OpenStack services should all be on the same machine.
  2. OpenStack web services should run on ports other than 443.

I think both of these are ideas who's time have come and gone.

If you have a single machine, run them out of separate containers. That allows different services to work with different versions of the libraries. It lets you mix a newer Keystone with older everything else.

Often the APIs are deployed on a common set of nodes.

If everything is on port 443, you need a single web server at the front end to multiplex it; uWSGI or any other one does not obviate that.

++

There are no good ports left in /etc/services; stop trying to reserve new ones for the web. If you need to run on a web service, you need to be able to get through firewalls. You need to run on standard ports. Run on 443.

Keystone again is a great example of this: it has two ports: 5000 and 35357.

port 5000 in /etc/services is

commplex-main 5000/tcp

and port 35357 is smack dab in the middle of the ephemeral range.

This is a disconnect between linux and IANA. IANA has said 35357 is not ephemeral, linux defaults to say it is.

Again, so long as the web server supports the cryptographic secure mechanisms, I don't care what one you chose. But The idea of use going to Keystone and getting a bearer token as the basis for security is immature; we should be doing the following on every call:

  1. TLS
  2. Cryptographic authentication.

They can be together or split up.

So, lets get everything running inside Apache, and, at the same time, push our other favorite web servers to support the necessary pieces to make OpenStack and the Web secure.

++. We should do this and also document alternatives for wsgi which has no impact on this goal. Lets try and keep focused on the different initiatives and not cross the reasons for them.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 25, 2015 by Morgan_Fainberg (17,320 points)   2 6 9
...