Question

Hello

What kind of storage backend do you guys use if you see disk IO bottlenecks
when storing ceilometer events and metrics? In my current configuration I
am using 300 GB 10K SAS (in hardware raid 1) and iostat report does not
look good (upto 100% unilization) with ceilometer consuming high CPU and
Memory. Does it help adding more spindles and move to raid 10?

Thanks!
Paras.

---

OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Hello

What kind of storage backend do you guys use if you see disk IO bottlenecks when storing ceilometer events and metrics? In my current configuration I am using 300 GB 10K SAS (in hardware raid 1) and iostat report does not look good (upto 100% unilization) with ceilometer consuming high CPU and Memory.  Does it help adding more spindles and move to raid 10?

Thanks!

Paras.

Answer 1

On 08/04/17 01:00 PM, Paras pradhan wrote:
y current configuration I am using 300 GB 10K SAS (in hardware raid 1)
and iostat report does not look good (upto 100% unilization) with
ceilometer consuming high CPU and Memory. Does it help adding more
spindles and move to raid 10?

i don't have any recommendations on how to improve legacy ceilometer
storage. internally, i have gnocchi deployed with redis as incoming
storage and ceph as aggregate storage and it's working pretty well. i
should probably add this requires master so you'd need to wait until
gnocchi v4.

cheers,

--
gord

---

OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

On 08/04/17 01:00 PM, Paras pradhan wrote:
> y current configuration I am using 300 GB 10K SAS (in hardware raid 1)
> and iostat report does not look good (upto 100% unilization) with
> ceilometer consuming high CPU and Memory.  Does it help adding more
> spindles and move to raid 10?
>

i don't have any recommendations on how to improve legacy ceilometer
storage. internally, i have gnocchi deployed with redis as incoming
storage and ceph as aggregate storage and it's working pretty well. i
should probably add this requires master so you'd need to wait until
gnocchi v4.

cheers,

--
gord

_______________________________________________
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Answer 2

Ceilometer can be a pain in the a* if not properly configured/designed,
especially when things start to grow. I've already saw the exact same
situation you described on two different instalations. To make things more
complicated, some OpenStack distributions use MongoDB as a storage backend
and do not consider a dedicated infrastructure for Ceilometer, relegating
this important service to live, by default, in the controller nodes...
worst: not clearly agreeing on what should be done when the service starts
to stall rather than simply adding more controller nodes... (yes Red Hat,
I'm looking to you). You might consider using gnocchi and a ceph storage
for telemetry as it was already suggested.

For my 2 cents, here's a nice talk on the matter:
https://www.openstack.org/videos/video/capacity-planning-saving-money-and-maximizing-efficiency-in-openstack-using-gnocchi-and-ceilometer

[]'s
Hubner

On Sat, Apr 8, 2017 at 2:00 PM, Paras pradhan pradhanparas@gmail.com
wrote:

Hello

What kind of storage backend do you guys use if you see disk IO
bottlenecks when storing ceilometer events and metrics? In my current
configuration I am using 300 GB 10K SAS (in hardware raid 1) and iostat
report does not look good (upto 100% unilization) with ceilometer consuming
high CPU and Memory. Does it help adding more spindles and move to raid 10?

Thanks!
Paras.

---

OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

---

OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Ceilometer can be a pain in the a* if not properly configured/designed, especially when things start to grow. I've already saw the exact same situation you described on two different instalations. To make things more complicated, some OpenStack distributions use MongoDB as a storage backend and do not consider a dedicated infrastructure for Ceilometer, relegating this important service to live, by default, in the controller nodes... worst: not clearly agreeing on what should be done when the service starts to stall rather than simply adding more controller nodes... (yes Red Hat, I'm looking to you). You might consider using gnocchi and a ceph storage for telemetry as it was already suggested.

For my 2 cents, here's a nice talk on the matter: https://www.openstack.org/videos/video/capacity-planning-saving-money-and-maximizing-efficiency-in-openstack-using-gnocchi-and-ceilometer

[]'s

Hubner

On Sat, Apr 8, 2017 at 2:00 PM, Paras pradhan <pradhanparas@gmail.com> wrote:

Hello

What kind of storage backend do you guys use if you see disk IO bottlenecks when storing ceilometer events and metrics? In my current configuration I am using 300 GB 10K SAS (in hardware raid 1) and iostat report does not look good (upto 100% unilization) with ceilometer consuming high CPU and Memory.  Does it help adding more spindles and move to raid 10?

Thanks!

Paras.

_______________________________________________
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Answer 3

Thanks for the reply. I have see people recommending to use ceph as a
backed for gnocchi but we don't use Ceph yet and looks like gnocchi does
not support cinder backends. We use Dell EQ San for block devices.

-Paras.

On Tue, Apr 11, 2017 at 9:15 PM, Alex Hubner alex@hubner.net.br wrote:

Ceilometer can be a pain in the a* if not properly configured/designed,
especially when things start to grow. I've already saw the exact same
situation you described on two different instalations. To make things more
complicated, some OpenStack distributions use MongoDB as a storage backend
and do not consider a dedicated infrastructure for Ceilometer, relegating
this important service to live, by default, in the controller nodes...
worst: not clearly agreeing on what should be done when the service starts
to stall rather than simply adding more controller nodes... (yes Red Hat,
I'm looking to you). You might consider using gnocchi and a ceph storage
for telemetry as it was already suggested.

For my 2 cents, here's a nice talk on the matter: https://www.openstack.
org/videos/video/capacity-planning-saving-money-and-
maximizing-efficiency-in-openstack-using-gnocchi-and-ceilometer

[]'s
Hubner

On Sat, Apr 8, 2017 at 2:00 PM, Paras pradhan pradhanparas@gmail.com
wrote:

Hello

What kind of storage backend do you guys use if you see disk IO
bottlenecks when storing ceilometer events and metrics? In my current
configuration I am using 300 GB 10K SAS (in hardware raid 1) and iostat
report does not look good (upto 100% unilization) with ceilometer consuming
high CPU and Memory. Does it help adding more spindles and move to raid 10?

Thanks!
Paras.

---

OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

---

OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Thanks for the reply. I have see people recommending to use ceph as a backed for gnocchi but we don't use Ceph yet and looks like gnocchi does not support cinder backends. We use Dell EQ San for block devices.

-Paras.

On Tue, Apr 11, 2017 at 9:15 PM, Alex Hubner <alex@hubner.net.br> wrote:

Ceilometer can be a pain in the a* if not properly configured/designed, especially when things start to grow. I've already saw the exact same situation you described on two different instalations. To make things more complicated, some OpenStack distributions use MongoDB as a storage backend and do not consider a dedicated infrastructure for Ceilometer, relegating this important service to live, by default, in the controller nodes... worst: not clearly agreeing on what should be done when the service starts to stall rather than simply adding more controller nodes... (yes Red Hat, I'm looking to you). You might consider using gnocchi and a ceph storage for telemetry as it was already suggested.

For my 2 cents, here's a nice talk on the matter: https://www.openstack.org/videos/video/capacity-planning-saving-money-and-maximizing-efficiency-in-openstack-using-gnocchi-and-ceilometer

[]'s

Hubner

On Sat, Apr 8, 2017 at 2:00 PM, Paras pradhan <pradhanparas@gmail.com> wrote:

Hello

What kind of storage backend do you guys use if you see disk IO bottlenecks when storing ceilometer events and metrics? In my current configuration I am using 300 GB 10K SAS (in hardware raid 1) and iostat report does not look good (upto 100% unilization) with ceilometer consuming high CPU and Memory.  Does it help adding more spindles and move to raid 10?

Thanks!

Paras.

_______________________________________________
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators