settingsLogin | Registersettings

[openstack-dev] [Octavia] Proposal to support multiple listeners on one HAProxy instance

0 votes

I am proposing that Octavia should support deployment models that
enable multiple listeners to be configured inside the HAProxy
instance.

The model I am proposing is:

  1. One or more VIP per Octavia VM (propose one VIP in 0.5 release)
  2. One or more HAProxy instance per Octavia VM
  3. One or more listeners on each HAProxy instance
  4. Zero or more pools per listener (shared pools should be supported
    as a configuration render optimization, but propose support post 0.5
    release)
  5. One or more members per pool

This provides flexibility to the operator to support multiple
deployment models, including active-active and hot standby Octavia
VMs. Without the flexibility to have multiple listeners per HAProxy
instance we are limiting the operators deployment models.

I am advocating for multiple listeners per HAProxy instance because I
think it provides the following advantages:

  1. It reduces memory overhead due to running multiple HAProxy
    instances on one Octavia VM. Since the Octavia constitution states
    that Octavia is for large operators where this memory overhead could
    have a financial impact we should allow alternate deployment options.
  2. It reduces host CPU overhead due to reduced context switching that
    would occur between HAProxy instances. HAProxy is event driven and
    will mostly be idle waiting for traffic, where multiple instances of
    HAProxy will require context switching between the processes which
    increases the VM?s CPU load. Since the Octavia constitution states
    that we are designing for large operators, anything we can do to
    reduce the host CPU load reduces the operator?s costs.
  3. Hosting multiple HAProxy instances on one Octavia VM will increase
    the load balancer build time because multiple configuration files,
    start/stop scripts, health monitors, and HAProxy Unix sockets will
    have to be created. This could significantly impact operator
    topologies that use hot standby Octavia VMs for failover.
  4. It reduces network traffic and health monitoring overhead because
    only one HAProxy instance per Octavia VM will need to be monitored.
    This again, saves the operator money and increases the scalability for
    large operators.
  5. Multiple listeners per instance allows the sharing of backend pools
    which reduces the amount of health monitoring traffic required to the
    backend servers. It also has the potential to share SSL certificates
    and keys.
  6. It allows customers to think of load balancers (floating IPs) as an
    application service, sharing the fate of multiple listeners and
    providing a consolidated log file. This also provides a natural
    grouping of services (around the floating IP) for a defined
    performance floor. With multiple instances per Octavia VM one
    instance could negatively impact all of the other instances which may
    or may not be related to the other floating IP(s).
  7. Multiple listeners per instance reduces the number of TCP ports
    used on the Octavia VM, increasing the per-VM scalability.

I don?t want us, by design, to limit the operator flexibility in
deployment an topologies, especially when it potentially impacts the
costs for large operators. Having multiple listeners per HAProxy
instance is a very common topology in the HAProxy community and I
don?t think we should block that use case with Octavia deployments.

Michael

asked Aug 20, 2014 in openstack-dev by Michael_Johnson (4,380 points)   4 7
retagged Feb 25, 2015 by admin

3 Responses

0 votes

Hi Michael!

Just to give others some background on this: The current proposal (by me)
is to have each Listener object, (as defined in the Neutron LBaaS v2 code
base) correspond with one haproxy process on the Octavia VM in the
currently proposed Octavia design document. Michael's proposal is to have
each Loadbalancer object correspond with one haproxy process (which would
have multiple front-end sections in it to service each Listener on the
Loadbalancer).

Anyway, we thought it would be useful to discuss this on the mailing list
so that we could give others a chance to register their opinions, and
justify the same.

That being said, my responses to your points are in-line below, followed by
my reasoning for wanting 1 haproxy process = 1 listener in the
implementation:

On Wed, Aug 20, 2014 at 12:34 PM, Michael Johnson
wrote:

I am proposing that Octavia should support deployment models that
enable multiple listeners to be configured inside the HAProxy
instance.

The model I am proposing is:

  1. One or more VIP per Octavia VM (propose one VIP in 0.5 release)
  2. One or more HAProxy instance per Octavia VM
  3. One or more listeners on each HAProxy instance

This is where our proposals differ. I propose 1 listener per haproxy
instance.

  1. Zero or more pools per listener (shared pools should be supported
    as a configuration render optimization, but propose support post 0.5
    release)
  2. One or more members per pool

I would also propose zero or more members per pool. A pool with zero
members in it has been (is being) used by some of our customers to
blacklist certain client IP addresses. These customers want to respond to
the blacklisted IPs with an error 503 page (which can be done by haproxy)
instead of simply not responding to packets (if the blacklist were done at
the firewall).

This provides flexibility to the operator to support multiple
deployment models, including active-active and hot standby Octavia
VMs. Without the flexibility to have multiple listeners per HAProxy
instance we are limiting the operators deployment models.

I don't think your conclusion follows logically from your justification
here. Specifically, active-active and hot standby Octavia VMs are equally
supported by a one-process-per-listener model. Further, for reasons I'll
get into below, I think the one-process-per-listener model actually
provides more flexibility to the operators and users in how services are
deployed. Therefore, the conclusion I come to is the exact opposite of
yours: By insisting that all listeners on a given loadbalancer share a
single haproxy process, we actually limit flexibility in deployment models
(as well as introduce some potential operational problems we otherwise
wouldn't encounter).

I am advocating for multiple listeners per HAProxy instance because I
think it provides the following advantages:

  1. It reduces memory overhead due to running multiple HAProxy
    instances on one Octavia VM. Since the Octavia constitution states
    that Octavia is for large operators where this memory overhead could
    have a financial impact we should allow alternate deployment options.

  2. It reduces host CPU overhead due to reduced context switching that
    would occur between HAProxy instances. HAProxy is event driven and
    will mostly be idle waiting for traffic, where multiple instances of
    HAProxy will require context switching between the processes which
    increases the VM?s CPU load. Since the Octavia constitution states
    that we are designing for large operators, anything we can do to
    reduce the host CPU load reduces the operator?s costs.

So these two points might be the only compelling reason I see to follow the
approach you suggest. However, I would like to see the savings here
justified via benchmarks. If benchmarks don't show a significant difference
in performance running multiple haproxy instances to service different
listeners over running a single haproxy instance servicing the same
listeners, then I don't think these points are sufficient justification. I
understand your team (HP) is going to be working on these, hopefully in
time for next week's Octavia meeting.

Please also understand that memory and CPU usage are just two factors in
determining overall cost of the solution. Slowing progress on delivering
features, increasing faults and other problems by having a more complicated
configuration, and making problems more difficult to isolate and
troubleshoot are also factors that affect the cost of a solution (though
they aren't as easy to quantify). Therefore it does not necessarily
logically follow that "anything" we can do to reduce CPU load decreases the
operator's costs.

Keep in mind, also, that for large operators the scaling strategy is to
ensure services can be scaled horizontally (meaning the CPU / memory
footprint of a single process isn't very important for a large load that
will be spread across many machines anyway), and any costs for delivering
the service are usually passed on to the user in a more-or-less direct
fashion. If a simpler solution is 5% less efficient than the more
complicated one, but experiences 50% fewer problems... as both an operator
and a user I'll pick the simpler solution.

  1. Hosting multiple HAProxy instances on one Octavia VM will increase
    the load balancer build time because multiple configuration files,
    start/stop scripts, health monitors, and HAProxy Unix sockets will
    have to be created. This could significantly impact operator
    topologies that use hot standby Octavia VMs for failover.

I disagree. I think you're making several assumptions about how Octavia VMs
will be configured once they are up and running which are not true.
Specifically:

  • Yes, one-haproxy-per-listener does require multiple configuration files.
    These typically take times measured in milliseconds to generate and deploy.
    This does not significantly impact the time to deploy new services.
  • I think you're assuming the start/stop scripts used to control the
    haproxy processes will be the ones shipped with standard OS haproxy
    packages. This will not be the case with Octavia VMs, and a single set of
    scripts for this purpose will work for all haproxy instances on a single VM.
  • Health monitors are built into haproxy itself. A multi-listener
    loadbalancer configuration using multiple pools will use the same number of
    health monitor threads no matter whether this is accomplished using
    multiple haproxy instances or just one.
  • HAproxy unix sockets are created at approximate the same rate that a
    process can create a file. In other words, tens of thousands per second.
    Plus using multiple processes just increases the number of unix sockets in
    use by the number of processes used (typically one or two in most
    configurations). I think you're really scraping for justification for your
    idea if you seriously think this is going to be a performance problem. :/

One point: It's true that if pools are shared between listeners there are
overall fewer healthmonitors that need to be set up for the same group of
back-end servers. But I don't think this is sufficient justification for
sharing between listeners because:

  1. In nearly all the configurations in use in our customer environments,
    there actually isn't any sharing of pools going on between listeners on a
    single 'loadbalancer' (for us 'VIP', but it's the same thing): Even in
    situations where HTTP and HTTPS services point to the same group of
    back-end servers, the HTTPS connections are usually being forwarded to a
    different port than the HTTP connections, which means these are effectively
    separate pools.
  2. If protocols mismatch (ex. 'HTTP' listener versus 'TCP' listener versus
    non-terminated 'HTTPS' listener), then pool sharing is not an option for
    obvious technical reasons.
  3. In the very few cases where pools could be shared between listeners it's
    almost always between HTTP and TERMINATED_HTTPS. In other words, just two
    listeners on a single loadbalancer (not the hypothetical and probably
    non-existent 50 listeners on the same loadbalancer pointing at the same
    pool).
  4. In the very few cases where pools could be shared between listeners,
    this means that the number of health checks going to the back-end is
    increased by a multiple of the number of haproxy processes servicing all
    the listeners. In other words, instead of getting one health-check per 5
    seconds, each back-end server gets two health-checks per 5 seconds. If the
    back-end server or network infrastructure can't handle this, then there's
    something seriously wrong with the application being load balanced or
    network infrastructure.
  1. It reduces network traffic and health monitoring overhead because
    only one HAProxy instance per Octavia VM will need to be monitored.
    This again, saves the operator money and increases the scalability for
    large operators.

The health-check of the Octavia VM will happen over a (single) REST API
request-response cycle that will look essentially no different if multiple
haproxy instances are used or if just one is. Also, in your model above, if
a single Octavia VM is being used to service multiple loadbalancers, then
your model would also involve multiple haproxy processes on a single
Octavia VM. So this point is invalid.

  1. Multiple listeners per instance allows the sharing of backend pools
    which reduces the amount of health monitoring traffic required to the
    backend servers. It also has the potential to share SSL certificates
    and keys.

I made my point about health monitors above... But just to put things in
perspective: A typical health check involves a single, simple HTTP request
response cycle every 5 seconds to each member of a given pool, and usually
entails at most a few kilobytes of data transfer (usually < 1KB). This is
on a system designed to handle gigabits per second of data throughput for
the production load. We're talking many orders of magnitude difference
between the health check traffic and the potential production load. Again,
I think you're really grasping at straws if you think this is a serious
point to consider in justifying your argument.

There's also no reason the one-haproxy-per-listener model can't share SSL
certificates and keys (though this is a fairly rare set-up among our
customer base. Almost always a single SSL certificate+key is going to be
used on one listener).

  1. It allows customers to think of load balancers (floating IPs) as an
    application service, sharing the fate of multiple listeners and
    providing a consolidated log file. This also provides a natural
    grouping of services (around the floating IP) for a defined
    performance floor. With multiple instances per Octavia VM one
    instance could negatively impact all of the other instances which may
    or may not be related to the other floating IP(s).

Consolidated log files can be used with multiple haproxy instances as well.

The 'defined performance floor' or grouping of services around a floating
IP (loadbalancer as defined in Neutron LBaaS v2) happens at the Octavia VM
level in either your proposal or mine, so this point is invalid.

With multiple listeners per haproxy instance, any one overly-used listener
(TCP port) will negatively affect the listeners on other ports as well--
so that's really no different than one haproxy instance per listener,
again, nullifying the point I think you were making.

I'm also not sure why you're bringing up other instances on other floating
IPs. Keep in mind that 'floating IP' here is the same thing as
'loadbalancer' as defined in the Neutron LBaaS v2 discussion. The model you
propose would also use multiple haproxy instances (one per floating IP),
and if you're worried about a bad neighbor problem on one floating IP
affecting services on another floating IP, you're problem isn't the number
of haproxy instances you're running, it's the number of floating IPs
(loadbalancers) you're allowing per Octavia VM. Our models do not differ at
all with regard to how to schedule loadbalancer deployments to Octavia VMs
at all, therefore... once again this point is invalid.

But there is one difference between your model and mine raised above. You
said that multiple listeners per haproxy process allows them to "sharing
the fate of multiple listeners." I think that you mean that if a problem
happens with Listener A, then Listener B (on the same loadbalancer) is also
affected because it's the same "service."

Said another way, you're saying that in your model there is no fault
isolation between listeners on a given loadbalancer. I think in IRC
discussions you touted this as a feature. I would instead say it's an
anti-feature: Not having fault isolation between listeners is actually a
bad thing that is avoided in my model.

I also don't see any reason why having multiple listeners per haproxy
instance versus one listener per haproxy instance would affect the way the
customer thinks about the service at all: After all, this information is
not (and should not be) exposed to the customer in any way.

  1. Multiple listeners per instance reduces the number of TCP ports
    used on the Octavia VM, increasing the per-VM scalability.

Um... no it doesn't. If you're serving the same load from the same clients
to the same back-end servers for the same front-end listeners... then all
things being equal, you're using the same number of TCP ports whether this
is accomplished using one haproxy process or multiple haproxy processes.
I'm confused as to why you would expect anything different.

I don?t want us, by design, to limit the operator flexibility in
deployment an topologies, especially when it potentially impacts the
costs for large operators. Having multiple listeners per HAProxy
instance is a very common topology in the HAProxy community and I
don?t think we should block that use case with Octavia deployments.

Right... so I think that we would actually be limiting operator flexibility
by insisting that a single haproxy instance be used to service multiple
listeners (ie. I come to the exact opposite conclusion using the same data
that you are). In your back-of-the-envelope calculations of cost you're
ignoring costs associated with longer times to develop new features given a
more complicated back-end, increased costs due to higher faults,
troubleshooting, and support costs for a more complicated back-end, etc.

Having multiple listeners per haproxy instance is a common topology in the
HAProxy community because it's the one people are subtly pushed toward
using by reading the haproxy configuration guide, and because they're using
OS-packaged control scripts, which are usually designed to allow for just
one haproxy process, because most servers being deployed are not meant to
be highly scalable, programmable load balancers. However, I think that one
haproxy process per floating IP is actually not an optimal choice for a VM
appliance that is intended to be thus.

Anyway, now that I've spent time poking holes in your justifications for a
multiple-listeners-per-haproxy implementation, let me add a few more to
think about as positive reasons for using a one-listener-per-haproxy
implementation:

A. Despite what you imply above at several points, one-listener-per-haproxy
does not reduce the flexibility of the solution from either the user's or
operator's perspective.

B. One-listener-per-haproxy actually increases the flexibility of the
solution because:
i. If the operator wants to use different log files per listener, they can
easily do so (but are not forced to do so).
ii. Certain haproxy configuration keywords are more logically expressed in
a the 'defaults' configuration area in the configuration file (since the
'listen' section won't be used in our templates-- only 'defaults'
'frontend' and 'backend'). Using one-listener-per-haproxy configuration
files, this allows us to easily vary parameters that would go in this
'defaults' section. In other words, different listeners can have different
haproxy 'defaults' easily. (This allows for a simple way to implement, say
a non-zero keepalive timeout for SSL termination, where keepalive can make
a huge performance difference, and no keepalive for a HTTP listener on the
same loadbalancer, where there are other performance benefits for doing
this.)

C. One-listener-per-haproxy configuration templates will be much simpler
than multiple-listener-per-haproxy templates, which has implications for
troubleshooting, feature development, test writing, and code maintenance
and flexibility. While less tangible, these all reduce the overall cost and
increase the reliability of the solution.

D. Fault isolation is a good thing. If a problem happens with some random
custom TCP listener on port 8686, it's better that this does not take out
the production HTTP listener on port 80 at the same time. This is a good
feature to have, not a bug.

E. Not only do we get fault isolation with this design, but we isolate
service disruptions due to normal updates to listeners, pools, or ports.
The only way to alter a running haproxy configuration is to restart the
haproxy process with a new configuration file. Haproxy provides for a way
to do this gracefully, but it still resets usage counters and health
monitor states for all listeners covered by that haproxy configuration.
And if there's a problem running the new configuration (such that an
attempted graceful restart brings the service down), all listeners are
affected. This is an unnecessary are unjustified risk since it's so easily
avoided. Why should my attempt to, for example, update a member
configuration on a pool associated with the HTTPS listener not only reset
the usage counters and potentially disrupt end-users currently using the
port 80 HTTP listener, but also risk breaking that service entirely if the
resultant configuration file doesn't run for some reason?

F. It's actually easier / simpler / more reliable to parse the usage
statistic data for a listener for haproxy if only one listener per haproxy
instance is used.

G. Usage statistics have more intuitive inherent meaning if pools can only
be shared within one listener's configuration. In other words, a pool's
usage statistics are naturally interpreted to mean 'within the context of
the parent listener'. (And, yes, the distinction between listener and pool
usage data becomes meaningful when L7 rules are used.)

H. The above point is also true for operational status information.

I. Since minimum resource guarantees happen at the Octavia VM level, then
SLAs are essentially equivalent using either your model or mine.

J. Troubleshooting is actually much easier to do with
one-listener-per-haproxy. For example:
i. An operator logged into an Octavia VM can easily see the different
resource usage of each listener on that VM using standard unix tools like
'top', 'ps', etc. This is not possible if all the listeners are being
serviced by a single haproxy instance.
ii. An operator can start / stop / manually alter the configuration for one
listener without affecting any of the other listeners on the Octavia VM.
Again, this is not possible if all listeners are serviced by one process.
iii. If desired, an Operator can easily alter global logging configuration
(alter log file location, increase verbosity, etc.) for a single listener
(or all listeners), as well as easily filter out log lines for unaffected
listeners if we go with the one-listener-per-haproxy model. This, again, is
not trivial to do on an individual listener basis using the
multiple-listeners-per-haproxy model.
iv. The same goes for any other parameters that logically fall into the
'defaults' section of the haproxy configuration.

And... I think that's about it.

I also don't want us, by design, to limit the operator flexibility in
deployment
and topologies, especially when it potentially impacts the costs for large
operators. I simply come to the exact opposite conclusion than you do when
using the above justifications. :)

Thanks,
Stephen

--
Stephen Balukoff
Blue Box Group, LLC
(800)613-4305 x807
-------------- next part --------------
An HTML attachment was scrubbed...
URL:

responded Aug 21, 2014 by Stephen_Balukoff (8,040 points)   1 5 6
0 votes

I'm on the fence here, I see a number of advantages to each:

Single HAProxy process per listener:

  • Failure isolation
  • TLS Performance -- for non TLS services HAProxy is IO bound, and there
    is no reason to run it across multiple CPU cores, but with HAProxy
    terminating TLS there is an increased potential of a DoS with a large
    number of incoming TLS handshakes.
  • Reduced impact of reconfiguration -- while there is very little impact
    when reloading the configuration since HAProxy hands off the listener
    sockets to the new instance and the old instance continues to handle those
    connections, with a more complex configuration it is more likely to affect
    services on other listeners.

Multiple listeners on a single HAProxy instance:

  • Enables sharing pools between listeners -- this would reduce keep
    health monitor traffic, and pipe-lining requests from multiple listeners is
    possible
  • Reduced resource usage -- we should preform the benchmarks and
    quantify this
  • Simplified stats/log aggregation
  • Simplified Octavia instances -- I think each Octavia instance only
    running a single HAProxy process is a win, its easier to monitor and
    upstart/systemd/init only needs to start a single process.

Dustin Lundquist
-------------- next part --------------
An HTML attachment was scrubbed...
URL:

responded Aug 21, 2014 by Dustin_Lundquist (580 points)   1
0 votes

Hi Dustin,

Responses in-line:

On Thu, Aug 21, 2014 at 1:56 PM, Dustin Lundquist
wrote:

I'm on the fence here, I see a number of advantages to each:

Single HAProxy process per listener:

  • Failure isolation
  • TLS Performance -- for non TLS services HAProxy is IO bound, and
    there is no reason to run it across multiple CPU cores, but with HAProxy
    terminating TLS there is an increased potential of a DoS with a large
    number of incoming TLS handshakes.
  • Reduced impact of reconfiguration -- while there is very little
    impact when reloading the configuration since HAProxy hands off the
    listener sockets to the new instance and the old instance continues to
    handle those connections, with a more complex configuration it is more
    likely to affect services on other listeners.

Multiple listeners on a single HAProxy instance:

  • Enables sharing pools between listeners -- this would reduce keep
    health monitor traffic, and pipe-lining requests from multiple listeners is
    possible

I spoke to this point above. Frankly, I'm starting to think this argument
might be premature optimization: I'm guessing the number of incidents
where pools are shared between listeners on a single loadbalancer is going
to be relatively rare-- so few as to not merit consideration for the
overall design. :/

  • Reduced resource usage -- we should preform the benchmarks and
    quantify this

Yep, I'm looking forward to seeing the benchmarks here.

  • Simplified stats/log aggregation

I disagree here. This is especially the case if we use something like
syslog-ng for gathering logs (which is effectively non-blocking, which is
probably desirable no matter whether one haproxy process or multiple
haproxy processes are used). I'm not sure haproxy's code for appending logs
it writes to directly is non-blocking.

Stats parsing from haproxy is simpler if more processes are used. As far as
aggregation: Well, we've yet to define what people might want aggregated.
But note here that shared pools across listeners means shared stats for
those pools: A user might want to see that pool's stats for listener A
versus listener B, which isn't possible if the pool is shared across
listeners. :/ (In any case, we're still talking hypotheticals here...)

  • Simplified Octavia instances -- I think each Octavia instance only
    running a single HAProxy process is a win, its easier to monitor and
    upstart/systemd/init only needs to start a single process.

So, in the model proposed by Michael, a single haproxy instance consists
of all the listeners on that loadbalancer as a single process. So if more
than one loadbalancer is deployed to a single Octavia VM, you're going to
need to start / stop / otherwise control multiple haproxy processes anyway.
So the system upstart / systemd / init scripts aren't going to cut it for
this set-up. My thought was to write a new control script (similar to the
one we use in our environment already) which controls all the haproxy
processes, and which can be called on boot to look for and start any
processes for which configuration exists locally (assuming persistent
storage for the VM or something-- if some operators want to do this). It's
just as likely that we would have a freshly-booted Octavia VM check in with
its controller on boot, download any configurations it should be running,
and start the associated haproxy process(es). Again, the model proposed by
Michael and the model proposed by me do not differ much in how this control
must work if we're allowing multiple loadbalancers per Octavia VM.

We can potentially debate whether we allow multiple loadbalancers per
Octavia VM, but I think restricting this to a maximum of one is not
desirable from a hardware utilization perspective. Many production load
balanced services sit nearly idle all day, so there's no reason an Operator
shouldn't be allowed to combine multiple loadbalancers on a single Octavia
VM (perhaps at a lower price tier to the user). This is also similar to how
actual load balancing hardware appliance vendors tend to operate. The
restrction of 1 loadbalancer per Octavia VM does limit the operator's
options, eh.

Stephen

--
Stephen Balukoff
Blue Box Group, LLC
(800)613-4305 x807
-------------- next part --------------
An HTML attachment was scrubbed...
URL:

responded Aug 21, 2014 by Stephen_Balukoff (8,040 points)   1 5 6
...