settingsLogin | Registersettings

Re: [openstack-dev] [Openstack-operators] [Ceilometer] Real world experience with Ceilometer deployments - Feedback requested

0 votes

hi,
just to follow-up, thanks for the input, the usability of ceilometer is obviously a concern of ours and something the team tries to address with the resources we have.
as a quick help/update, here are some points of interests that i think might help:- if using Juno+, DO use the notifier:// publisher rather than rpc:// as there is a certain level of overhead that comes with rpc[1]. you can also configure multiple messaging servers if there are load issues.- a part of the telemetry team has been exploring tsdb and we expect to have a tech preview for Kilo. the project is called Gnocchi[2]- in Kilo, we expanded notification event handling (existing stacktach integration code) and said events can be published to an external source(s) or to a database (ElasticSearch for full-text querying, in addition to mongo, sql)- ceilometer does not configure databases. operators are expected to read up on the db of choice and properly configure db to their needs (ie. don't run default mongo install on a single node with no sharding to store data from 2000 nodes)[3]- DO adjust your pipeline to only store events/meters that you use. by default, ceilometer gives you the world and from there you can filter based on requirements.- it's entirely possible to use ceilometer to gather data and store it externally and avoid ceilometer storage (if you so choose)- DO NOT use SQL backend prior to Juno... for any deployment size... any...- there was some work in Kilo to jitter polling cycle of agents to distribute load.- the agents are designed to scale horizontally to increase bandwidth. also, they work independently so if you want just notifications, it's possible to just deploy the notification agent and nothing else.
we've also been updating -- and still continuing to update -- some of the docs to better reflect some of the changes made to Ceilometer in Juno and Kilo[4][5]. particularly, i'd probably look at the architecture diagram[6] to get an idea of what components of ceilometer you could use to fit your needs.
i'm probably missed stuff but i hope the above helps. as always, community help is always invited. if you have a patch that will improve ceilometer, the community gladly welcomes it.
[1] https://www.rabbitmq.com/tutorials/tutorial-six-python.html[2] http://www.slideshare.net/EoghanGlynn/rdo-hangout-on-gnocchi [3] http://blog.sileht.net/using-a-shardingreplicaset-mongodb-with-ceilometer[4] http://docs.openstack.org/admin-guide-cloud/content/ch_admin-openstack-telemetry.html[5] http://docs.openstack.org/developer/ceilometer/[6] http://docs.openstack.org/developer/ceilometer/architecture.html (self-plug for my amazing diagram skills)
cheers,
gord
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

asked Mar 10, 2015 in openstack-operators by gordon_chung (19,300 points)   2 3 12
retagged Mar 19, 2015 by admin

5 Responses

0 votes

sorry, i apparently don't know how to format emails...

cheers,
gord

From: gord@live.ca
To: openstack-operators@lists.openstack.org; openstack-dev@lists.openstack.org
Date: Tue, 10 Mar 2015 16:05:47 -0400
Subject: Re: [openstack-dev] [Openstack-operators] [Ceilometer] Real world experience with Ceilometer deployments - Feedback requested

hi,
just to follow-up, thanks for the input, the usability of ceilometer is obviously a concern of ours and something the team tries to address with the resources we have.
as a quick help/update, here are some points of interests that i think might help:- if using Juno+, DO use the notifier:// publisher rather than rpc:// as there is a certain level of overhead that comes with rpc[1]. you can also configure multiple messaging servers if there are load issues.- a part of the telemetry team has been exploring tsdb and we expect to have a tech preview for Kilo. the project is called Gnocchi[2]- in Kilo, we expanded notification event handling (existing stacktach integration code) and said events can be published to an external source(s) or to a database (ElasticSearch for full-text querying, in addition to mongo, sql)- ceilometer does not configure databases. operators are expected to read up on the db of choice and properly configure db to their needs (ie. don't run default mongo install on a single node with no sharding to store data from 2000 nodes)[3]- DO adjust your pipeline to only store events/meters that you use. by default, ceilometer gives you the world and from there you can filter based on requirements.- it's entirely possible to use ceilometer to gather data and store it externally and avoid ceilometer storage (if you so choose)- DO NOT use SQL backend prior to Juno... for any deployment size... any...- there was some work in Kilo to jitter polling cycle of agents to distribute load.- the agents are designed to scale horizontally to increase bandwidth. also, they work independently so if you want just notifications, it's possible to just deploy the notification agent and nothing else.
we've also been updating -- and still continuing to update -- some of the docs to better reflect some of the changes made to Ceilometer in Juno and Kilo[4][5]. particularly, i'd probably look at the architecture diagram[6] to get an idea of what components of ceilometer you could use to fit your needs.
i'm probably missed stuff but i hope the above helps. as always, community help is always invited. if you have a patch that will improve ceilometer, the community gladly welcomes it.
[1] https://www.rabbitmq.com/tutorials/tutorial-six-python.html[2] http://www.slideshare.net/EoghanGlynn/rdo-hangout-on-gnocchi [3] http://blog.sileht.net/using-a-shardingreplicaset-mongodb-with-ceilometer[4] http://docs.openstack.org/admin-guide-cloud/content/ch_admin-openstack-telemetry.html[5] http://docs.openstack.org/developer/ceilometer/[6] http://docs.openstack.org/developer/ceilometer/architecture.html (self-plug for my amazing diagram skills)
cheers,
gord


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev _______________________________________________
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
responded Mar 10, 2015 by gordon_chung (19,300 points)   2 3 12
0 votes

On 03/10/2015 04:34 PM, gordon chung wrote:
sorry, i apparently don't know how to format emails...

I actually like the format, we should all switch to outlook.
Kidding.

Just bringing my two cents about MongoDB sharding, which is changing
performances at scale; though the ElasticSearch is also very
interesting. Is it something that is gated in OpenStack Infra?
Can we expect something usable in production for Kilo?

cheers,
/gord/


From: gord@live.ca
To: openstack-operators@lists.openstack.org;
openstack-dev@lists.openstack.org
Date: Tue, 10 Mar 2015 16:05:47 -0400
Subject: Re: [openstack-dev] [Openstack-operators] [Ceilometer] Real
world experience with Ceilometer deployments - Feedback requested

hi,

just to follow-up, thanks for the input, the usability of ceilometer is
obviously a concern of ours and something the team tries to address with
the resources we have.

as a quick help/update, here are some points of interests that i think
might help:
- if using Juno+, DO use the notifier:// publisher rather than rpc:// as
there is a certain level of overhead that comes with rpc[1]. you can
also configure multiple messaging servers if there are load issues.
- a part of the telemetry team has been exploring tsdb and we expect to
have a tech preview for Kilo. the project is called Gnocchi[2]
- in Kilo, we expanded notification event handling (existing stacktach
integration code) and said events can be published to an external
source(s) or to a database (ElasticSearch for full-text querying, in
addition to mongo, sql)
- ceilometer does not configure databases. operators are expected to
read up on the db of choice and properly configure db to their needs
(ie. don't run default mongo install on a single node with no sharding
to store data from 2000 nodes)[3]
- DO adjust your pipeline to only store events/meters that you use. by
default, ceilometer gives you the world and from there you can filter
based on requirements.
- it's entirely possible to use ceilometer to gather data and store it
externally and avoid ceilometer storage (if you so choose)
- DO NOT use SQL backend prior to Juno... for any deployment size... any...
- there was some work in Kilo to jitter polling cycle of agents to
distribute load.
- the agents are designed to scale horizontally to increase bandwidth.
also, they work independently so if you want just notifications, it's
possible to just deploy the notification agent and nothing else.

we've also been updating -- and still continuing to update -- some of
the docs to better reflect some of the changes made to Ceilometer in
Juno and Kilo[4][5]. particularly, i'd probably look at the architecture
diagram[6] to get an idea of what components of ceilometer you could use
to fit your needs.

i'm probably missed stuff but i hope the above helps. as always,
community help is always invited. if you have a patch that will improve
ceilometer, the community gladly welcomes it.

[1] https://www.rabbitmq.com/tutorials/tutorial-six-python.html
[2] http://www.slideshare.net/EoghanGlynn/rdo-hangout-on-gnocchi
[3] http://blog.sileht.net/using-a-shardingreplicaset-mongodb-with-ceilometer
[4] http://docs.openstack.org/admin-guide-cloud/content/ch_admin-openstack-telemetry.html
[5] http://docs.openstack.org/developer/ceilometer/
[6] http://docs.openstack.org/developer/ceilometer/architecture.html (self-plug
for my amazing diagram skills)

cheers,
/gord/


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

--
Emilien Macchi


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

responded Mar 10, 2015 by emilien_at_redhat.co (36,940 points)   3 8 16
0 votes

I actually like the format, we should all switch to outlook.
Kidding.

all the cool kids are doing it... i suggest you add your year of birth to the end to be unique.

Is it something that is gated in OpenStack Infra?
Can we expect something usable in production for Kilo?

ElasticSearch (or 'elastic' as they apparently just rebranded today), is in the process of being gated on[1] and will be available in Kilo.

[1] http://review.openstack.org/#/c/160824/

cheers,
gord


Date: Tue, 10 Mar 2015 17:10:24 -0400
From: emilien@redhat.com
To: openstack-operators@lists.openstack.org
Subject: Re: [Openstack-operators] [openstack-dev] [Ceilometer] Real world experience with Ceilometer deployments - Feedback requested

On 03/10/2015 04:34 PM, gordon chung wrote:
sorry, i apparently don't know how to format emails...

I actually like the format, we should all switch to outlook.
Kidding.

Just bringing my two cents about MongoDB sharding, which is changing
performances at scale; though the ElasticSearch is also very
interesting. Is it something that is gated in OpenStack Infra?
Can we expect something usable in production for Kilo?

cheers,
/gord/


From: gord@live.ca
To: openstack-operators@lists.openstack.org;
openstack-dev@lists.openstack.org
Date: Tue, 10 Mar 2015 16:05:47 -0400
Subject: Re: [openstack-dev] [Openstack-operators] [Ceilometer] Real
world experience with Ceilometer deployments - Feedback requested

hi,

just to follow-up, thanks for the input, the usability of ceilometer is
obviously a concern of ours and something the team tries to address with
the resources we have.

as a quick help/update, here are some points of interests that i think
might help:
- if using Juno+, DO use the notifier:// publisher rather than rpc:// as
there is a certain level of overhead that comes with rpc[1]. you can
also configure multiple messaging servers if there are load issues.
- a part of the telemetry team has been exploring tsdb and we expect to
have a tech preview for Kilo. the project is called Gnocchi[2]
- in Kilo, we expanded notification event handling (existing stacktach
integration code) and said events can be published to an external
source(s) or to a database (ElasticSearch for full-text querying, in
addition to mongo, sql)
- ceilometer does not configure databases. operators are expected to
read up on the db of choice and properly configure db to their needs
(ie. don't run default mongo install on a single node with no sharding
to store data from 2000 nodes)[3]
- DO adjust your pipeline to only store events/meters that you use. by
default, ceilometer gives you the world and from there you can filter
based on requirements.
- it's entirely possible to use ceilometer to gather data and store it
externally and avoid ceilometer storage (if you so choose)
- DO NOT use SQL backend prior to Juno... for any deployment size... any...
- there was some work in Kilo to jitter polling cycle of agents to
distribute load.
- the agents are designed to scale horizontally to increase bandwidth.
also, they work independently so if you want just notifications, it's
possible to just deploy the notification agent and nothing else.

we've also been updating -- and still continuing to update -- some of
the docs to better reflect some of the changes made to Ceilometer in
Juno and Kilo[4][5]. particularly, i'd probably look at the architecture
diagram[6] to get an idea of what components of ceilometer you could use
to fit your needs.

i'm probably missed stuff but i hope the above helps. as always,
community help is always invited. if you have a patch that will improve
ceilometer, the community gladly welcomes it.

[1] https://www.rabbitmq.com/tutorials/tutorial-six-python.html
[2] http://www.slideshare.net/EoghanGlynn/rdo-hangout-on-gnocchi
[3] http://blog.sileht.net/using-a-shardingreplicaset-mongodb-with-ceilometer
[4] http://docs.openstack.org/admin-guide-cloud/content/ch_admin-openstack-telemetry.html
[5] http://docs.openstack.org/developer/ceilometer/
[6] http://docs.openstack.org/developer/ceilometer/architecture.html (self-plug
for my amazing diagram skills)

cheers,
/gord/


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

--
Emilien Macchi


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
responded Mar 10, 2015 by gordon_chung (19,300 points)   2 3 12
0 votes

On 03/10/2015 06:18 PM, gordon chung wrote:

I actually like the format, we should all switch to outlook.
Kidding.

all the cool kids are doing it... i suggest you add your year of birth to the end to be unique.

My first e-mail was actually something like yours, but with my year of
birth at the end, what a coincidence !

Is it something that is gated in OpenStack Infra?
Can we expect something usable in production for Kilo?

ElasticSearch (or 'elastic' as they apparently just rebranded today), is in the process of being gated on[1] and will be available in Kilo.

[1] http://review.openstack.org/#/c/160824/

Looking at the patch, so "experimental" in gate config means this
feature will be experimental in Kilo, and we should not expect something
stable & tested enough for large scale deployment in production. Right?

cheers,
gord


Date: Tue, 10 Mar 2015 17:10:24 -0400
From: emilien@redhat.com
To: openstack-operators@lists.openstack.org
Subject: Re: [Openstack-operators] [openstack-dev] [Ceilometer] Real world experience with Ceilometer deployments - Feedback requested

On 03/10/2015 04:34 PM, gordon chung wrote:

sorry, i apparently don't know how to format emails...

I actually like the format, we should all switch to outlook.
Kidding.

Just bringing my two cents about MongoDB sharding, which is changing
performances at scale; though the ElasticSearch is also very
interesting. Is it something that is gated in OpenStack Infra?
Can we expect something usable in production for Kilo?

cheers,
/gord/


From: gord@live.ca
To: openstack-operators@lists.openstack.org;
openstack-dev@lists.openstack.org
Date: Tue, 10 Mar 2015 16:05:47 -0400
Subject: Re: [openstack-dev] [Openstack-operators] [Ceilometer] Real
world experience with Ceilometer deployments - Feedback requested

hi,

just to follow-up, thanks for the input, the usability of ceilometer is
obviously a concern of ours and something the team tries to address with
the resources we have.

as a quick help/update, here are some points of interests that i think
might help:
- if using Juno+, DO use the notifier:// publisher rather than rpc:// as
there is a certain level of overhead that comes with rpc[1]. you can
also configure multiple messaging servers if there are load issues.
- a part of the telemetry team has been exploring tsdb and we expect to
have a tech preview for Kilo. the project is called Gnocchi[2]
- in Kilo, we expanded notification event handling (existing stacktach
integration code) and said events can be published to an external
source(s) or to a database (ElasticSearch for full-text querying, in
addition to mongo, sql)
- ceilometer does not configure databases. operators are expected to
read up on the db of choice and properly configure db to their needs
(ie. don't run default mongo install on a single node with no sharding
to store data from 2000 nodes)[3]
- DO adjust your pipeline to only store events/meters that you use. by
default, ceilometer gives you the world and from there you can filter
based on requirements.
- it's entirely possible to use ceilometer to gather data and store it
externally and avoid ceilometer storage (if you so choose)
- DO NOT use SQL backend prior to Juno... for any deployment size... any...
- there was some work in Kilo to jitter polling cycle of agents to
distribute load.
- the agents are designed to scale horizontally to increase bandwidth.
also, they work independently so if you want just notifications, it's
possible to just deploy the notification agent and nothing else.

we've also been updating -- and still continuing to update -- some of
the docs to better reflect some of the changes made to Ceilometer in
Juno and Kilo[4][5]. particularly, i'd probably look at the architecture
diagram[6] to get an idea of what components of ceilometer you could use
to fit your needs.

i'm probably missed stuff but i hope the above helps. as always,
community help is always invited. if you have a patch that will improve
ceilometer, the community gladly welcomes it.

[1] https://www.rabbitmq.com/tutorials/tutorial-six-python.html
[2] http://www.slideshare.net/EoghanGlynn/rdo-hangout-on-gnocchi
[3] http://blog.sileht.net/using-a-shardingreplicaset-mongodb-with-ceilometer
[4] http://docs.openstack.org/admin-guide-cloud/content/ch_admin-openstack-telemetry.html
[5] http://docs.openstack.org/developer/ceilometer/
[6] http://docs.openstack.org/developer/ceilometer/architecture.html (self-plug
for my amazing diagram skills)

cheers,
/gord/


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

--
Emilien Macchi


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

--
Emilien Macchi


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

responded Mar 11, 2015 by emilien_at_redhat.co (36,940 points)   3 8 16
0 votes

Looking at the patch, so "experimental" in gate config means this
feature will be experimental in Kilo, and we should not expect something
stable & tested enough for large scale deployment in production. Right?

the experimental tag is mainly because it's a new check and i don't want to implode the gate. but yes, it is a new backend and new features so it's something that still needs to be vetted over time. once we're sure it's stable in gate and stable outside gate we can make it more permanent.

cheers,
gord

Date: Tue, 10 Mar 2015 23:01:12 -0400
From: emilien@redhat.com
To: gord@live.ca; openstack-operators@lists.openstack.org
Subject: Re: [Openstack-operators] [openstack-dev] [Ceilometer] Real world experience with Ceilometer deployments - Feedback requested

On 03/10/2015 06:18 PM, gordon chung wrote:

I actually like the format, we should all switch to outlook.
Kidding.

all the cool kids are doing it... i suggest you add your year of birth to the end to be unique.

My first e-mail was actually something like yours, but with my year of
birth at the end, what a coincidence !

Is it something that is gated in OpenStack Infra?
Can we expect something usable in production for Kilo?

ElasticSearch (or 'elastic' as they apparently just rebranded today), is in the process of being gated on[1] and will be available in Kilo.

[1] http://review.openstack.org/#/c/160824/

Looking at the patch, so "experimental" in gate config means this
feature will be experimental in Kilo, and we should not expect something
stable & tested enough for large scale deployment in production. Right?

cheers,
gord


Date: Tue, 10 Mar 2015 17:10:24 -0400
From: emilien@redhat.com
To: openstack-operators@lists.openstack.org
Subject: Re: [Openstack-operators] [openstack-dev] [Ceilometer] Real world experience with Ceilometer deployments - Feedback requested

On 03/10/2015 04:34 PM, gordon chung wrote:

sorry, i apparently don't know how to format emails...

I actually like the format, we should all switch to outlook.
Kidding.

Just bringing my two cents about MongoDB sharding, which is changing
performances at scale; though the ElasticSearch is also very
interesting. Is it something that is gated in OpenStack Infra?
Can we expect something usable in production for Kilo?

cheers,
/gord/


From: gord@live.ca
To: openstack-operators@lists.openstack.org;
openstack-dev@lists.openstack.org
Date: Tue, 10 Mar 2015 16:05:47 -0400
Subject: Re: [openstack-dev] [Openstack-operators] [Ceilometer] Real
world experience with Ceilometer deployments - Feedback requested

hi,

just to follow-up, thanks for the input, the usability of ceilometer is
obviously a concern of ours and something the team tries to address with
the resources we have.

as a quick help/update, here are some points of interests that i think
might help:
- if using Juno+, DO use the notifier:// publisher rather than rpc:// as
there is a certain level of overhead that comes with rpc[1]. you can
also configure multiple messaging servers if there are load issues.
- a part of the telemetry team has been exploring tsdb and we expect to
have a tech preview for Kilo. the project is called Gnocchi[2]
- in Kilo, we expanded notification event handling (existing stacktach
integration code) and said events can be published to an external
source(s) or to a database (ElasticSearch for full-text querying, in
addition to mongo, sql)
- ceilometer does not configure databases. operators are expected to
read up on the db of choice and properly configure db to their needs
(ie. don't run default mongo install on a single node with no sharding
to store data from 2000 nodes)[3]
- DO adjust your pipeline to only store events/meters that you use. by
default, ceilometer gives you the world and from there you can filter
based on requirements.
- it's entirely possible to use ceilometer to gather data and store it
externally and avoid ceilometer storage (if you so choose)
- DO NOT use SQL backend prior to Juno... for any deployment size... any...
- there was some work in Kilo to jitter polling cycle of agents to
distribute load.
- the agents are designed to scale horizontally to increase bandwidth.
also, they work independently so if you want just notifications, it's
possible to just deploy the notification agent and nothing else.

we've also been updating -- and still continuing to update -- some of
the docs to better reflect some of the changes made to Ceilometer in
Juno and Kilo[4][5]. particularly, i'd probably look at the architecture
diagram[6] to get an idea of what components of ceilometer you could use
to fit your needs.

i'm probably missed stuff but i hope the above helps. as always,
community help is always invited. if you have a patch that will improve
ceilometer, the community gladly welcomes it.

[1] https://www.rabbitmq.com/tutorials/tutorial-six-python.html
[2] http://www.slideshare.net/EoghanGlynn/rdo-hangout-on-gnocchi
[3] http://blog.sileht.net/using-a-shardingreplicaset-mongodb-with-ceilometer
[4] http://docs.openstack.org/admin-guide-cloud/content/ch_admin-openstack-telemetry.html
[5] http://docs.openstack.org/developer/ceilometer/
[6] http://docs.openstack.org/developer/ceilometer/architecture.html (self-plug
for my amazing diagram skills)

cheers,
/gord/


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

--
Emilien Macchi


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

--
Emilien Macchi

_______________________________________________
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

responded Mar 11, 2015 by gordon_chung (19,300 points)   2 3 12
...