settingsLogin | Registersettings

[Openstack-operators] Nodes and configurations management in Puppet

0 votes

Hi,

Some of you use Puppet to manage your OpenStack infrastructure.

  • How do you manage your node definitions?
    Do you have an external ENC?
    Or plain site.pp, Puppet Enterprise, theforeman, etc. ?

  • How about your configuration?
    Do you use Hiera? Or do you rely on the ENC to manage them?

My question is related to the complexity that managing multiple
OpenStack environments (staging/production), regions and cells involves
over time.

Is there a magically way to manage node definitions and especially
configurations so you guys no have a heart attack each time you have to
dig into them? How about versioning?

To answer my own questions and start the discussion:

I don't use an external ENC. The site.pp manifest has been the one used
since day one. Since we have a strong host naming convention, I didn't
see the limit of this model (yet). Regex has been a good friend so far.

As for configurations, Hiera is used to organize then with a hierarchy
to manage environments and regions specific configurations:

  • "environments/%{::environment}/regions/%{::openstack_region}/common"
  • "environments/%{::environment}/common"
  • common

I'm still exploring solutions for cells.

How about you guys?

--
Mathieu

asked Sep 25, 2014 in openstack-operators by Mathieu_Gagné (3,300 points)   1 3 6
retagged Apr 14, 2015 by admin

10 Responses

0 votes

Hi Mathieu,

My setup is very similar to yours. Node definitions are in site.pp and
Hiera is used for all configuration. The Hiera hierarchies are also very
similar.

Overall, I have a love/hate relationship with the setup. I could go on in
detail, but it'd be all Puppet-specific rather than OpenStack. I'd be happy
to discuss off-list.

Of if there's enough interest, I can post it here. I just don't want to
muddy up this list with non-OpenStack things.

Thanks,
Joe

On Thu, Sep 25, 2014 at 8:40 AM, Mathieu Gagn? wrote:

Hi,

Some of you use Puppet to manage your OpenStack infrastructure.

  • How do you manage your node definitions?
    Do you have an external ENC?
    Or plain site.pp, Puppet Enterprise, theforeman, etc. ?

  • How about your configuration?
    Do you use Hiera? Or do you rely on the ENC to manage them?

My question is related to the complexity that managing multiple OpenStack
environments (staging/production), regions and cells involves over time.

Is there a magically way to manage node definitions and especially
configurations so you guys no have a heart attack each time you have to dig
into them? How about versioning?

To answer my own questions and start the discussion:

I don't use an external ENC. The site.pp manifest has been the one used
since day one. Since we have a strong host naming convention, I didn't see
the limit of this model (yet). Regex has been a good friend so far.

As for configurations, Hiera is used to organize then with a hierarchy to
manage environments and regions specific configurations:

  • "environments/%{::environment}/regions/%{::openstack_region}/common"
  • "environments/%{::environment}/common"
  • common

I'm still exploring solutions for cells.

How about you guys?

--
Mathieu


OpenStack-operators mailing list
OpenStack-operators at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

-------------- next part --------------
An HTML attachment was scrubbed...
URL:

responded Sep 25, 2014 by Joe_Topjian (5,780 points)   1 4 7
0 votes

We have a single default node definition. We have a custom fact that
determines the node's role based on it's hostname, then includes a derived
class ("include role::${::role}") based off of that information. Since
we're using something very similar to Craig Dunn's roles/profiles pattern,
and we store all the configuration in hiera, I feel like we're getting most
of the benefits of an ENC without having to maintain one.

Our hiera hierarchy is pretty gross right now, since it's layered on top of
what came with puppetopenstackbuilder originally. We don't use puppet
environments. Since we have a puppet master per site & environment, we
haven't had a need for them yet and I'm afraid of this bug:
http://projects.puppetlabs.com/issues/12173

We do have a concept of "echelon" in our hierarchy, which functionally is
identical to what most people would call "environment". We looked for a
synonym for environment to avoid overloading the term. Right now we have
'prod', 'staging' and 'dev' echelons and we can specify values by role
inside of each echelon also.

We also have a site level in our hierarchy that we can again break out per
role, but usually don't.

Lastly, we have a role level in our hierarchy and a main "common" sort of
hierarchy. The role hierarchy is mostly used to keep role specific
configuration out of the main "common" file since that's pretty large. We
do sometimes use it for overriding the common hierarchy.

As far as versioning, we have two long lived branches, "prod" and
"master". Master is managed by Gerrit and we do the normal pre-merge
testing that you'd expect. Right now the prod branch is managed more
manually. When we want to do a deployment prepare a "proposed" branch that
should be able to fast-forward merge on to the prod branch, although we
always force a merge commit for documentation purposes. The proposed
branch is usually just a snapshot of the master branch at a point in time,
but sometimes it's more targeted for specific fixes.

On Thu, Sep 25, 2014 at 7:40 AM, Mathieu Gagn? wrote:

Hi,

Some of you use Puppet to manage your OpenStack infrastructure.

  • How do you manage your node definitions?
    Do you have an external ENC?
    Or plain site.pp, Puppet Enterprise, theforeman, etc. ?

  • How about your configuration?
    Do you use Hiera? Or do you rely on the ENC to manage them?

My question is related to the complexity that managing multiple OpenStack
environments (staging/production), regions and cells involves over time.

Is there a magically way to manage node definitions and especially
configurations so you guys no have a heart attack each time you have to dig
into them? How about versioning?

To answer my own questions and start the discussion:

I don't use an external ENC. The site.pp manifest has been the one used
since day one. Since we have a strong host naming convention, I didn't see
the limit of this model (yet). Regex has been a good friend so far.

As for configurations, Hiera is used to organize then with a hierarchy to
manage environments and regions specific configurations:

  • "environments/%{::environment}/regions/%{::openstack_region}/common"
  • "environments/%{::environment}/common"
  • common

I'm still exploring solutions for cells.

How about you guys?

--
Mathieu


OpenStack-operators mailing list
OpenStack-operators at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

-------------- next part --------------
An HTML attachment was scrubbed...
URL:

responded Sep 25, 2014 by Clayton_O'Neill (2,060 points)   2
0 votes

On 09/25/2014 11:45 AM, Joe Topjian wrote:
Hi Mathieu,

My setup is very similar to yours. Node definitions are in site.pp and
Hiera is used for all configuration. The Hiera hierarchies are also very
similar.

Overall, I have a love/hate relationship with the setup. I could go on
in detail, but it'd be all Puppet-specific rather than OpenStack. I'd be
happy to discuss off-list.

Of if there's enough interest, I can post it here. I just don't want to
muddy up this list with non-OpenStack things.

I certainly would enjoy a good discussion here about your love-hate
relationship with your deployment tooling. I have the same relationship
with the Chef work we did at AT&T and am curious whether there are
Puppetisms that similarly cause grief as some Chefisms.

/me readies the popcorn.

Best,
-jay

responded Sep 25, 2014 by Jay_Pipes (59,760 points)   3 10 14
0 votes

-----Original Message-----
From: Jay Pipes [mailto:jaypipes at gmail.com]
Sent: 25 September 2014 20:12
To: openstack-operators at lists.openstack.org
Subject: Re: [Openstack-operators] Nodes and configurations management in
Puppet

On 09/25/2014 11:45 AM, Joe Topjian wrote:

Hi Mathieu,

My setup is very similar to yours. Node definitions are in site.pp and
Hiera is used for all configuration. The Hiera hierarchies are also
very similar.

Overall, I have a love/hate relationship with the setup. I could go on
in detail, but it'd be all Puppet-specific rather than OpenStack. I'd
be happy to discuss off-list.

Of if there's enough interest, I can post it here. I just don't want
to muddy up this list with non-OpenStack things.

I certainly would enjoy a good discussion here about your love-hate relationship
with your deployment tooling. I have the same relationship with the Chef work
we did at AT&T and am curious whether there are Puppetisms that similarly
cause grief as some Chefisms.

/me readies the popcorn.

There is also puppet-openstack at puppetlabs.com which is where the Puppet/OpenStack team hang out.

Tim

Best,
-jay


OpenStack-operators mailing list
OpenStack-operators at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
responded Sep 25, 2014 by Tim_Bell (16,440 points)   1 6 10
0 votes

On 9/25/14 7:40 AM, "Mathieu Gagn?" wrote:

Hi,

Some of you use Puppet to manage your OpenStack infrastructure.

  • How do you manage your node definitions?
    Do you have an external ENC?
    Or plain site.pp, Puppet Enterprise, theforeman, etc. ?

We manage node definitions using hostnames, the role of the node and some
information about geography and echelon (dev, staging, prod, etc) is
encoded within it. The basic data that gets the node booted and installed
is YAML that puppet turns into cobbler config.

  • How about your configuration?
    Do you use Hiera? Or do you rely on the ENC to manage them?

My question is related to the complexity that managing multiple
OpenStack environments (staging/production), regions and cells involves
over time.

Is there a magically way to manage node definitions and especially
configurations so you guys no have a heart attack each time you have to
dig into them? How about versioning?

We version all our config and hiera data with git on multiple branches.

To answer my own questions and start the discussion:

I don't use an external ENC. The site.pp manifest has been the one used
since day one. Since we have a strong host naming convention, I didn't
see the limit of this model (yet). Regex has been a good friend so far.

We came up with the host naming convention before we brought up a single
node. Custom facter facts parses the hostname into information like role,
region, etc, that then determines the hiera hierarchy. Clayton wrote most
of that code.

Honestly our hiera stack is probably too complex, but we have the ability
to customize almost every level in something like this simplified view of
our stack:

  • hostname
  • site/role
  • role
  • site
  • echelon/role
  • echelon
  • twc-common
  • common

There's a geography layer in there too that I did not show. For example,
if you had dev and staging in the same physical site they would share NTP
configs but not Keystone configs, so we try to move common stuff down as
low as possible.

As for configurations, Hiera is used to organize then with a hierarchy
to manage environments and regions specific configurations:

  • "environments/%{::environment}/regions/%{::openstack_region}/common"
  • "environments/%{::environment}/common"
  • common

I'm still exploring solutions for cells.

How about you guys?

I'd be happy to discuss this with you more any time.

--
Mathieu


OpenStack-operators mailing list
OpenStack-operators at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

This E-mail and any of its attachments may contain Time Warner Cable proprietary information, which is privileged, confidential, or subject to copyright belonging to Time Warner Cable. This E-mail is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient of this E-mail, you are hereby notified that any dissemination, distribution, copying, or action taken in relation to the contents of and attachments to this E-mail is strictly prohibited and may be unlawful. If you have received this E-mail in error, please notify the sender immediately and permanently delete the original and any copy of this E-mail and any printout.

responded Sep 25, 2014 by Fischer,_Matt (1,380 points)   1 3
0 votes

Hi Jay,

Hiera takes the cake for my love/hate of Puppet. I try really hard to keep
the number of hierarchies small and even then I find it awkward sometimes.
I love the concept of Hiera, but I find it can be unintuitive. Similar to
the other replies, I have a "common" hierarchy where 90% of the data is
stored. The other hierarchies either override "common" or append to it.
When I need to know where a parameter is ultimately configured, I find
myself thinking "is that parameter common across everything or specific to
a certain location or node, and if so, why did I make it specific?", then
doing a "grep -R" to find where it's located, and finally thinking "oh
right - that's why it's there".

Another area of Puppet that I'm finding difficult to work with is
configuring HA environments. There are two main pain points here and
they're pretty applicable to using Puppet with OpenStack:

The first is standing up a cluster of servers. For example MySQL and
Galera. The first server in the cluster gets a blank gcomm setting. This
means that I either need to have a blank gcomm setting in Hiera just for
the first node or I need to remove the list of nodes from Hiera temporarily
and add it back after the first node is set up. puppetdbquery
https://github.com/dalen/puppet-puppetdbquery (very awesome module) can
also be used. There are pros and cons to all of these.

After the first node, the rest of the servers are trickier because Puppet
will usually start the MySQL service before all settings are in place. Then
after all settings are in place, MySQL will restart. But the timing of
ensuring all settings are in place for the new node to connect to the other
nodes and sync with Galera is rarely right. This is fixable by doing a lot
of forced ordering in Puppet, but (at least in my current setup), that
would require a custom MySQL module or modifying the PuppetLabs MySQL
module.

And finally, just coordinating Puppet runs of multiple cluster members is a
bear. It can take up to 3 Puppet runs for two nodes to connect. Think syn,
syn/ack, ack here.

The other HA pain point is creating many-to-one configurations, for example
putting those MySQL nodes behind HAProxy. I want HAProxy to have an entry
for MySQL that lists all nodes. Listing the nodes is easy: each MySQL
server can simply export itself as a valid node, but what about the rest of
the HAProxy MySQL config that's "singular" (for example, the "listen" and
"option" settings)? Only one node can host that data or Puppet will
complain about duplicate data. I don't want one node responsible for the
data because I feel that creates a "pet" environment vs a "cattle" one.
There are two hacks I've done to get around this:

The first is to create resources in each MySQL server with unique resource
names but the same value. A simple example is:

file_line { '/etc/foobar.conf for node-1':
path => '/etc/foobar.conf',
line => 'Hello, World!',
}

file_line { '/etc/foobar.conf for node-2':
path => '/etc/foobar.conf',
line => 'Hello, World!',
}

Now if these resources are exported and collected on a central server,
Puppet will not complain about duplicate resources and the single line will
exist in the foobar.conf file.

The second hack is more involved and it would take a lot of detail to show
a full example. In summary, the Puppet stdlib module has a function
called ensureresource
https://github.com/puppetlabs/puppetlabs-stdlib#ensure_resourcethat will
only apply a resource if it doesn't already exist. If several nodes export
the same ensure
resource with the exact same settings, the single resource
will be built on the server that imports them. The problem here is that if
the resource is ever modified, then I must comment out the resource, run
"puppet apply --noop" so it's removed from PuppetDB, make the change, then
run Puppet again. If not, then the first node that runs with the new
setting will export the modified resource and Puppet will complain because
two resources exist with different data. This might sound very confusing
and awkward and it is, but doing this allows me to store resources like
firewall rules, MySQL users, HAProxy entries, etc on multiple nodes when
those settings should only be applied once in the entire environment.

I think a cleaner way of doing this is to introduce service discovery into
my environment, but I haven't had time to look into this in more detail.

I should mention that some of these HA pains can be resolved by just moving
all of the data to the HAProxy nodes themselves. So when I want to add a
new service, such as RabbitMQ, to HAProxy, I add the RabbitMQ settings to
the HAProxy role/profiles. But I want HAProxy to be "dumb" about what it's
hosting. I want to be able to use it in a Juju-like fashion where I can
introduce any arbitrary service and HAProxy configures itself without prior
knowledge of the new service.

I say some pains, because this isn't applicable to, say, a MySQL cluster
where users and databases are shared across the entire cluster.

There are probably one or two other areas I'm forgetting, but typing the
above has cleared my head. :)

In general, though, I really enjoy working with Puppet. Our current Puppet
configurations allow us to stand up test OpenStack environments with little
manual input as well as upgrade to newer releases of OpenStack with very
little effort. For example, short of our compute nodes, all other OpenStack
components run in LXC containers. Our Icehouse upgrade procedure consisted
of killing those containers, making new ones based on Ubuntu 14.04, and
letting Puppet do the rest.

Let me know if you'd like more details or clarification.

Thanks,
Joe

On Thu, Sep 25, 2014 at 12:12 PM, Jay Pipes wrote:

On 09/25/2014 11:45 AM, Joe Topjian wrote:

Hi Mathieu,

My setup is very similar to yours. Node definitions are in site.pp and
Hiera is used for all configuration. The Hiera hierarchies are also very
similar.

Overall, I have a love/hate relationship with the setup. I could go on
in detail, but it'd be all Puppet-specific rather than OpenStack. I'd be
happy to discuss off-list.

Of if there's enough interest, I can post it here. I just don't want to
muddy up this list with non-OpenStack things.

I certainly would enjoy a good discussion here about your love-hate
relationship with your deployment tooling. I have the same relationship
with the Chef work we did at AT&T and am curious whether there are
Puppetisms that similarly cause grief as some Chefisms.

/me readies the popcorn.

Best,
-jay


OpenStack-operators mailing list
OpenStack-operators at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

-------------- next part --------------
An HTML attachment was scrubbed...
URL:

responded Sep 26, 2014 by Joe_Topjian (5,780 points)   1 4 7
0 votes

On Thursday, September 25, 2014 at 7:30 PM, Joe Topjian wrote:
Hi Jay,

Hiera takes the cake for my love/hate of Puppet. I try really hard to keep the number of hierarchies small and even then I find it awkward sometimes. I love the concept of Hiera, but I find it can be unintuitive. Similar to the other replies, I have a "common" hierarchy where 90% of the data is stored. The other hierarchies either override "common" or append to it. When I need to know where a parameter is ultimately configured, I find myself thinking "is that parameter common across everything or specific to a certain location or node, and if so, why did I make it specific?", then doing a "grep -R" to find where it's located, and finally thinking "oh right - that's why it's there".

Another area of Puppet that I'm finding difficult to work with is configuring HA environments. There are two main pain points here and they're pretty applicable to using Puppet with OpenStack:

The first is standing up a cluster of servers. For example MySQL and Galera. The first server in the cluster gets a blank gcomm setting. This means that I either need to have a blank gcomm setting in Hiera just for the first node or I need to remove the list of nodes from Hiera temporarily and add it back after the first node is set up. puppetdbquery (https://github.com/dalen/puppet-puppetdbquery) (very awesome module) can also be used. There are pros and cons to all of these.

After the first node, the rest of the servers are trickier because Puppet will usually start the MySQL service before all settings are in place. Then after all settings are in place, MySQL will restart. But the timing of ensuring all settings are in place for the new node to connect to the other nodes and sync with Galera is rarely right. This is fixable by doing a lot of forced ordering in Puppet, but (at least in my current setup), that would require a custom MySQL module or modifying the PuppetLabs MySQL module.

And finally, just coordinating Puppet runs of multiple cluster members is a bear. It can take up to 3 Puppet runs for two nodes to connect. Think syn, syn/ack, ack here.
I know this sounds like crazy talk, but I actually use ansible here to orchestrate puppet/chef. The eventual consistency model really bugs me, and probably a source of ~60% of my problems. I personally find it easier to grok an over state file[1].

I much prefer ansible controlling most everything including the ancillary openstack services (rabbitmq, percona, etc..), and relying on the solid stackforge chef/puppet community simply for openstack. Again, I know I?m crazy.

[1] https://github.com/blueboxgroup/ursula/blob/master/site.yml


The other HA pain point is creating many-to-one configurations, for example putting those MySQL nodes behind HAProxy. I want HAProxy to have an entry for MySQL that lists all nodes. Listing the nodes is easy: each MySQL server can simply export itself as a valid node, but what about the rest of the HAProxy MySQL config that's "singular" (for example, the "listen" and "option" settings)? Only one node can host that data or Puppet will complain about duplicate data. I don't want one node responsible for the data because I feel that creates a "pet" environment vs a "cattle" one. There are two hacks I've done to get around this:

The first is to create resources in each MySQL server with unique resource names but the same value. A simple example is:

fileline { '/etc/foobar.conf for node-1':
path => '/etc/foobar.conf',
line => 'Hello, World!',
}

file
line { '/etc/foobar.conf for node-2':
path => '/etc/foobar.conf',
line => 'Hello, World!',
}


Now if these resources are exported and collected on a central server, Puppet will not complain about duplicate resources and the single line will exist in the foobar.conf file.

The second hack is more involved and it would take a lot of detail to show a full example. In summary, the Puppet stdlib module has a function called ensureresource (https://github.com/puppetlabs/puppetlabs-stdlib#ensure_resource)that will only apply a resource if it doesn't already exist. If several nodes export the same ensureresource with the exact same settings, the single resource will be built on the server that imports them. The problem here is that if the resource is ever modified, then I must comment out the resource, run "puppet apply --noop" so it's removed from PuppetDB, make the change, then run Puppet again. If not, then the first node that runs with the new setting will export the modified resource and Puppet will complain because two resources exist with different data. This might sound very confusing and awkward and it is, but doing this allows me to store resources like firewall rules, MySQL users, HAProxy entries, etc on multiple nodes when those settings should only be applied once in the entire environment.

I think a cleaner way of doing this is to introduce service discovery into my environment, but I haven't had time to look into this in more detail.

I should mention that some of these HA pains can be resolved by just moving all of the data to the HAProxy nodes themselves. So when I want to add a new service, such as RabbitMQ, to HAProxy, I add the RabbitMQ settings to the HAProxy role/profiles. But I want HAProxy to be "dumb" about what it's hosting. I want to be able to use it in a Juju-like fashion where I can introduce any arbitrary service and HAProxy configures itself without prior knowledge of the new service.

I say some pains, because this isn't applicable to, say, a MySQL cluster where users and databases are shared across the entire cluster.

There are probably one or two other areas I'm forgetting, but typing the above has cleared my head. :)

In general, though, I really enjoy working with Puppet. Our current Puppet configurations allow us to stand up test OpenStack environments with little manual input as well as upgrade to newer releases of OpenStack with very little effort. For example, short of our compute nodes, all other OpenStack components run in LXC containers. Our Icehouse upgrade procedure consisted of killing those containers, making new ones based on Ubuntu 14.04, and letting Puppet do the rest.

Let me know if you'd like more details or clarification.

Thanks,
Joe


On Thu, Sep 25, 2014 at 12:12 PM, Jay Pipes wrote:

On 09/25/2014 11:45 AM, Joe Topjian wrote:

Hi Mathieu,

My setup is very similar to yours. Node definitions are in site.pp and
Hiera is used for all configuration. The Hiera hierarchies are also very
similar.

Overall, I have a love/hate relationship with the setup. I could go on
in detail, but it'd be all Puppet-specific rather than OpenStack. I'd be
happy to discuss off-list.

Of if there's enough interest, I can post it here. I just don't want to
muddy up this list with non-OpenStack things.

I certainly would enjoy a good discussion here about your love-hate relationship with your deployment tooling. I have the same relationship with the Chef work we did at AT&T and am curious whether there are Puppetisms that similarly cause grief as some Chefisms.

/me readies the popcorn.

Best,
-jay



OpenStack-operators mailing list
OpenStack-operators at lists.openstack.org (mailto:OpenStack-operators at lists.openstack.org)
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


OpenStack-operators mailing list
OpenStack-operators at lists.openstack.org (mailto:OpenStack-operators at lists.openstack.org)
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

-------------- next part --------------
An HTML attachment was scrubbed...
URL:

responded Sep 26, 2014 by John_Dewey (1,320 points)   2
0 votes

Hi Clayton,

Thanks for sharing your experience and use of Puppet with OpenStack.

I feel less alone now. :)

On 2014-09-25 1:42 PM, Clayton O'Neill wrote:
We have a single default node definition. We have a custom fact that
determines the node's role based on it's hostname, then includes a
derived class ("include role::${::role}") based off of that
information. Since we're using something very similar to Craig Dunn's
roles/profiles pattern, and we store all the configuration in hiera, I
feel like we're getting most of the benefits of an ENC without having to
maintain one.

That's one hell of a clever idea you got there! :D

Our hiera hierarchy is pretty gross right now, since it's layered on top
of what came with puppetopenstackbuilder originally. We don't use
puppet environments. Since we have a puppet master per site &
environment, we haven't had a need for them yet and I'm afraid of this
bug: http://projects.puppetlabs.com/issues/12173

Thanks for pointing that one out. I didn't know about that particular bug.

--
Mathieu

responded Sep 26, 2014 by Mathieu_Gagné (3,300 points)   1 3 6
0 votes

Hi Joe,

Your experience and story about Puppet and OpenStack makes me feel like
you are a long lost co-worker. :)

On 2014-09-25 10:30 PM, Joe Topjian wrote:

Hiera takes the cake for my love/hate of Puppet. I try really hard to
keep the number of hierarchies small and even then I find it awkward
sometimes. I love the concept of Hiera, but I find it can be
unintuitive.

Same here. The aspect I hate about Hiera is that files become very big
and unorganized very fast due to the quantity of configs. So you try to
split them in multiples files instead and then you have the problem you
describe below...

Similar to the other replies, I have a "common" hierarchy
where 90% of the data is stored. The other hierarchies either override
"common" or append to it. When I need to know where a parameter is
ultimately configured, I find myself thinking "is that parameter common
across everything or specific to a certain location or node, and if so,
why did I make it specific?", then doing a "grep -R" to find where it's
located, and finally thinking "oh right - that's why it's there".

Yep. That's the feeling I was referring to when I said "heart attack".

And now, try to form a new co-worker and explain him how it's organized:
"Oh, I felt the file was too big so I split it in a hope to restore
sanity which it did with limited success."

The other difficulty is the management of "common" configs like keystone
auth URL. Multiple services need this value, yet their might be split in
multiple files and the YAML anchor hack [1] I used so far does not work
across YAML files. Same for database configs which are needed by the
database server (to provision the user) and services (for the database
connection string).

Another area of Puppet that I'm finding difficult to work with is
configuring HA environments. There are two main pain points here and
they're pretty applicable to using Puppet with OpenStack:

The other HA pain point is creating many-to-one configurations [...]

I think a cleaner way of doing this is to introduce service discovery
into my environment, but I haven't had time to look into this in more
detail.

I wholly agree with you and that's a concept I'm interested to explore.
Come to think of it, it strangely looks like the "dependency inversion
principle" in software development.

I however feel that an external ENC becomes inevitable to achieve this
ease of use. Unfortunately, each time I looked into it, I rapidly get
lost in my dream of a simple dashboard to manage everything. I feel I
rapidly come to the limits of what exported resources, Hiera and
puppetdb can do.

One idea would be to export an haproxy::listen resource from one of the
controller (which now becomes a pet as you said) and realize it on the
HAProxy nodes with its associated haproxy::member resources.

I should mention that some of these HA pains can be resolved by just
moving all of the data to the HAProxy nodes themselves. So when I want
to add a new service, such as RabbitMQ, to HAProxy, I add the RabbitMQ
settings to the HAProxy role/profiles. But I want HAProxy to be "dumb"
about what it's hosting. I want to be able to use it in a Juju-like
fashion where I can introduce any arbitrary service and HAProxy
configures itself without prior knowledge of the new service.

Yes! How do you guys think we can implement such discovery?

With Nova cells, this problem became much more apparent due to
inter-relations between the API cell and compute cells. The API cell has
to know about the compute cells and vice versa.

In general, though, I really enjoy working with Puppet. Our current
Puppet configurations allow us to stand up test OpenStack environments
with little manual input as well as upgrade to newer releases of
OpenStack with very little effort.

Yes, I really enjoy Puppet too. After all hardware/infrastructure
aspects are figured out, we are able to bootstrap a new OpenStack region
in less than an hour.

To summarize my current pain points:
- Out of control Hiera configuration files
- Lack of service auto-discovery

[1] https://dmsimard.com/2014/02/15/quick-hiera-tips/

--
Mathieu

responded Sep 26, 2014 by Mathieu_Gagné (3,300 points)   1 3 6
0 votes

On Sat, Sep 27, 2014 at 4:03 AM, Mathieu Gagn? wrote:

Hi Joe,

Your experience and story about Puppet and OpenStack makes me feel like
you are a long lost co-worker. :)

On 2014-09-25 10:30 PM, Joe Topjian wrote:

Hiera takes the cake for my love/hate of Puppet. I try really hard to
keep the number of hierarchies small and even then I find it awkward
sometimes. I love the concept of Hiera, but I find it can be
unintuitive.

Same here. The aspect I hate about Hiera is that files become very big and
unorganized very fast due to the quantity of configs. So you try to split
them in multiples files instead and then you have the problem you describe
below...

Similar to the other replies, I have a "common" hierarchy

where 90% of the data is stored. The other hierarchies either override
"common" or append to it. When I need to know where a parameter is
ultimately configured, I find myself thinking "is that parameter common
across everything or specific to a certain location or node, and if so,
why did I make it specific?", then doing a "grep -R" to find where it's
located, and finally thinking "oh right - that's why it's there".

Yep. That's the feeling I was referring to when I said "heart attack".

And now, try to form a new co-worker and explain him how it's organized:
"Oh, I felt the file was too big so I split it in a hope to restore sanity
which it did with limited success."

The other difficulty is the management of "common" configs like keystone
auth URL. Multiple services need this value, yet their might be split in
multiple files and the YAML anchor hack [1] I used so far does not work
across YAML files. Same for database configs which are needed by the
database server (to provision the user) and services (for the database
connection string).

Another area of Puppet that I'm finding difficult to work with is

configuring HA environments. There are two main pain points here and
they're pretty applicable to using Puppet with OpenStack:

The other HA pain point is creating many-to-one configurations [...]

I think a cleaner way of doing this is to introduce service discovery
into my environment, but I haven't had time to look into this in more
detail.

I wholly agree with you and that's a concept I'm interested to explore.
Come to think of it, it strangely looks like the "dependency inversion
principle" in software development.

I however feel that an external ENC becomes inevitable to achieve this
ease of use. Unfortunately, each time I looked into it, I rapidly get lost
in my dream of a simple dashboard to manage everything. I feel I rapidly
come to the limits of what exported resources, Hiera and puppetdb can do.

One idea would be to export an haproxy::listen resource from one of the
controller (which now becomes a pet as you said) and realize it on the
HAProxy nodes with its associated haproxy::member resources.

I should mention that some of these HA pains can be resolved by just

moving all of the data to the HAProxy nodes themselves. So when I want
to add a new service, such as RabbitMQ, to HAProxy, I add the RabbitMQ
settings to the HAProxy role/profiles. But I want HAProxy to be "dumb"
about what it's hosting. I want to be able to use it in a Juju-like
fashion where I can introduce any arbitrary service and HAProxy
configures itself without prior knowledge of the new service.

Yes! How do you guys think we can implement such discovery?

With Nova cells, this problem became much more apparent due to
inter-relations between the API cell and compute cells. The API cell has to
know about the compute cells and vice versa.

In general, though, I really enjoy working with Puppet. Our current

Puppet configurations allow us to stand up test OpenStack environments
with little manual input as well as upgrade to newer releases of
OpenStack with very little effort.

Yes, I really enjoy Puppet too. After all hardware/infrastructure aspects
are figured out, we are able to bootstrap a new OpenStack region in less
than an hour.

To summarize my current pain points:
- Out of control Hiera configuration files
- Lack of service auto-discovery

[1] https://dmsimard.com/2014/02/15/quick-hiera-tips/

--
Mathieu


OpenStack-operators mailing list
OpenStack-operators at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

I am working towards a client side ENC that queries a Consul [1] k/v store,
and building similar scripting to what I built for puppetopenstackbuilder
so that I can import hiera -> consul and set ACLs + node classifications.
An ENC would also avoid the 'everything must be a string' issues in hiera,
although maybe bodepd will finally solve that with this [2]. Consul already
has a decent dashboard (though not as nice as the puppet enterprise one) so
I should be able to leverage that.

Consul might similarly provide part of a suitable mechanism for discovery,
although the nature of Puppet and its expectation that data is available at
catalog compile time is a significant barrier to implementing this sanely,
since functions and facts are evaluated before resources, and resources
don't have a way to feed data back. My thoughts to mitigate this were to
split into smaller profiles and allow nodes to signal each other indicating
that a specific profile should be state-checked via running puppet, so that
I could watch consul and listen for messages and apply new data quickly,
but splitting catalogs sensibly is quite difficult.

I have a talk scheduled for the Paris summit on how I'm doing that using
Serf for the message bit, but I'm still not completely convinced it's
actually the best way to go about things, and I have a long way to go still
on implementation. Serf/Consul are also very new pieces of software so I'm
sure I'll hit more issues as I go.

I have heard there are others attempting similar things with etcd.

[1] http://www.consul.io/
[2] https://github.com/puppetlabs/hiera/pull/213
-------------- next part --------------
An HTML attachment was scrubbed...
URL:

responded Sep 27, 2014 by Michael_Chapman (720 points)   1
...