settingsLogin | Registersettings

[Openstack-operators] Ceilometer and disk IO

0 votes

Hello

What kind of storage backend do you guys use if you see disk IO bottlenecks
when storing ceilometer events and metrics? In my current configuration I
am using 300 GB 10K SAS (in hardware raid 1) and iostat report does not
look good (upto 100% unilization) with ceilometer consuming high CPU and
Memory. Does it help adding more spindles and move to raid 10?

Thanks!
Paras.


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
asked Apr 13, 2017 in openstack-operators by Paras_pradhan (1,080 points)   2

3 Responses

0 votes

On 08/04/17 01:00 PM, Paras pradhan wrote:
y current configuration I am using 300 GB 10K SAS (in hardware raid 1)
and iostat report does not look good (upto 100% unilization) with
ceilometer consuming high CPU and Memory. Does it help adding more
spindles and move to raid 10?

i don't have any recommendations on how to improve legacy ceilometer
storage. internally, i have gnocchi deployed with redis as incoming
storage and ceph as aggregate storage and it's working pretty well. i
should probably add this requires master so you'd need to wait until
gnocchi v4.

cheers,

--
gord


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
responded Apr 11, 2017 by gordon_chung (19,300 points)   2 3 7
0 votes

Ceilometer can be a pain in the a* if not properly configured/designed,
especially when things start to grow. I've already saw the exact same
situation you described on two different instalations. To make things more
complicated, some OpenStack distributions use MongoDB as a storage backend
and do not consider a dedicated infrastructure for Ceilometer, relegating
this important service to live, by default, in the controller nodes...
worst: not clearly agreeing on what should be done when the service starts
to stall rather than simply adding more controller nodes... (yes Red Hat,
I'm looking to you). You might consider using gnocchi and a ceph storage
for telemetry as it was already suggested.

For my 2 cents, here's a nice talk on the matter:
https://www.openstack.org/videos/video/capacity-planning-saving-money-and-maximizing-efficiency-in-openstack-using-gnocchi-and-ceilometer

[]'s
Hubner

On Sat, Apr 8, 2017 at 2:00 PM, Paras pradhan pradhanparas@gmail.com
wrote:

Hello

What kind of storage backend do you guys use if you see disk IO
bottlenecks when storing ceilometer events and metrics? In my current
configuration I am using 300 GB 10K SAS (in hardware raid 1) and iostat
report does not look good (upto 100% unilization) with ceilometer consuming
high CPU and Memory. Does it help adding more spindles and move to raid 10?

Thanks!
Paras.


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
responded Apr 12, 2017 by Alex_Hübner (200 points)  
0 votes

Thanks for the reply. I have see people recommending to use ceph as a
backed for gnocchi but we don't use Ceph yet and looks like gnocchi does
not support cinder backends. We use Dell EQ San for block devices.

-Paras.

On Tue, Apr 11, 2017 at 9:15 PM, Alex Hubner alex@hubner.net.br wrote:

Ceilometer can be a pain in the a* if not properly configured/designed,
especially when things start to grow. I've already saw the exact same
situation you described on two different instalations. To make things more
complicated, some OpenStack distributions use MongoDB as a storage backend
and do not consider a dedicated infrastructure for Ceilometer, relegating
this important service to live, by default, in the controller nodes...
worst: not clearly agreeing on what should be done when the service starts
to stall rather than simply adding more controller nodes... (yes Red Hat,
I'm looking to you). You might consider using gnocchi and a ceph storage
for telemetry as it was already suggested.

For my 2 cents, here's a nice talk on the matter: https://www.openstack.
org/videos/video/capacity-planning-saving-money-and-
maximizing-efficiency-in-openstack-using-gnocchi-and-ceilometer

[]'s
Hubner

On Sat, Apr 8, 2017 at 2:00 PM, Paras pradhan pradhanparas@gmail.com
wrote:

Hello

What kind of storage backend do you guys use if you see disk IO
bottlenecks when storing ceilometer events and metrics? In my current
configuration I am using 300 GB 10K SAS (in hardware raid 1) and iostat
report does not look good (upto 100% unilization) with ceilometer consuming
high CPU and Memory. Does it help adding more spindles and move to raid 10?

Thanks!
Paras.


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
responded Apr 13, 2017 by Paras_pradhan (1,080 points)   2
...