settingsLogin | Registersettings

[Openstack] [ceilometer] Troubleshooting Ceilometer

0 votes

We currently have a working Liberty based Openstack deployment based
on Ubuntu LT 14.04 (2 controllers, 4 compute).

We have followed the instructions at
http://docs.openstack.org/liberty/install-guide-ubuntu/ceilometer-install.html
and everything seems to have installed correctly. No errors in the
logs, and nothing really obviously wrong.

However in the instructions there is this page
http://docs.openstack.org/liberty/install-guide-ubuntu/ceilometer-verify.html
that shows the output of meter-list, however our meter-list output is
blank.

Example output:

$ ceilometer meter-list
+--------------+-------+-------+--------------------------------------+---------+------------+
| Name | Type | Unit | Resource ID
| User ID | Project ID |
+--------------+-------+-------+--------------------------------------+---------+------------+
| image | gauge | image | acafc7c0-40aa-4026-9673-b879898e1fc2
| None | cf12a15... |
| image.size | gauge | B | acafc7c0-40aa-4026-9673-b879898e1fc2
| None | cf12a15... |
+--------------+-------+-------+--------------------------------------+---------+------------+

root@controller01:/var/log/ceilometer# ceilometer meter-list
+------+------+------+-------------+---------+------------+
| Name | Type | Unit | Resource ID | User ID | Project ID |
+------+------+------+-------------+---------+------------+
+------+------+------+-------------+---------+------------+

There are no errors output, and it seems the command completes correctly.

So where am I going wrong here? Is there a step I am missing? Or is
there something obvious I have missed?

We have enabled glance, nova-compute, and cinder from the
instructions, but we aren't seeing any data (at least according to the
instructions).

Tom Walsh
Expresshosting.net


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
asked Oct 16, 2017 in openstack by expresswebsys_at_gma (340 points)   1 1

8 Responses

0 votes

Hi Tom,

You may check several things:
1. Do you have messages in queues and do the queues have listeners? For
example, check collector-related queue 'metering.sample' by default.

rabbitmqctl list_queues name messages consumers | grep metering

rabbitmqctl list_queues name messages consumers | grep notification

The both queues should have consumers

  1. Check entries directly in db you use.

Thanks,

Nadya

On Wed, Mar 2, 2016 at 6:57 AM, Tom Walsh expresswebsys@gmail.com wrote:

We currently have a working Liberty based Openstack deployment based
on Ubuntu LT 14.04 (2 controllers, 4 compute).

We have followed the instructions at

http://docs.openstack.org/liberty/install-guide-ubuntu/ceilometer-install.html
and everything seems to have installed correctly. No errors in the
logs, and nothing really obviously wrong.

However in the instructions there is this page

http://docs.openstack.org/liberty/install-guide-ubuntu/ceilometer-verify.html
that shows the output of meter-list, however our meter-list output is
blank.

Example output:

$ ceilometer meter-list

+--------------+-------+-------+--------------------------------------+---------+------------+
| Name | Type | Unit | Resource ID
| User ID | Project ID |

+--------------+-------+-------+--------------------------------------+---------+------------+
| image | gauge | image | acafc7c0-40aa-4026-9673-b879898e1fc2
| None | cf12a15... |
| image.size | gauge | B | acafc7c0-40aa-4026-9673-b879898e1fc2
| None | cf12a15... |

+--------------+-------+-------+--------------------------------------+---------+------------+

root@controller01:/var/log/ceilometer# ceilometer meter-list
+------+------+------+-------------+---------+------------+
| Name | Type | Unit | Resource ID | User ID | Project ID |
+------+------+------+-------------+---------+------------+
+------+------+------+-------------+---------+------------+

There are no errors output, and it seems the command completes correctly.

So where am I going wrong here? Is there a step I am missing? Or is
there something obvious I have missed?

We have enabled glance, nova-compute, and cinder from the
instructions, but we aren't seeing any data (at least according to the
instructions).

Tom Walsh
Expresshosting.net


Mailing list:
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe :
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
responded Mar 2, 2016 by Nadya_Privalova (1,640 points)   1 3
0 votes

Nadya,

Okay. So looking at these two here is what I am seeing:

metering.sample 0 1

notifications.error 5 0
notifications.info 1139 0
notifications.sample 18488 0

So I am guessing that means we have a consumer for metering, but not a
consumer on notifications (and there is a bit of a backlog on those
queues). Is that a correct statement?

We are utilizing the mongodb backend for storage, but we aren't overly
familiar with it or how to inspect data within it. Any pointers for
checking that side of things?

On Wed, Mar 2, 2016 at 5:10 AM, Nadya Shakhat nprivalova@mirantis.com wrote:
Hi Tom,

You may check several things:
1. Do you have messages in queues and do the queues have listeners? For
example, check collector-related queue 'metering.sample' by default.

rabbitmqctl list_queues name messages consumers | grep metering

rabbitmqctl list_queues name messages consumers | grep notification

The both queues should have consumers

  1. Check entries directly in db you use.

Thanks,

Nadya

On Wed, Mar 2, 2016 at 6:57 AM, Tom Walsh expresswebsys@gmail.com wrote:

We currently have a working Liberty based Openstack deployment based
on Ubuntu LT 14.04 (2 controllers, 4 compute).

We have followed the instructions at

http://docs.openstack.org/liberty/install-guide-ubuntu/ceilometer-install.html
and everything seems to have installed correctly. No errors in the
logs, and nothing really obviously wrong.

However in the instructions there is this page

http://docs.openstack.org/liberty/install-guide-ubuntu/ceilometer-verify.html
that shows the output of meter-list, however our meter-list output is
blank.

Example output:

$ ceilometer meter-list

+--------------+-------+-------+--------------------------------------+---------+------------+
| Name | Type | Unit | Resource ID
| User ID | Project ID |

+--------------+-------+-------+--------------------------------------+---------+------------+
| image | gauge | image | acafc7c0-40aa-4026-9673-b879898e1fc2
| None | cf12a15... |
| image.size | gauge | B | acafc7c0-40aa-4026-9673-b879898e1fc2
| None | cf12a15... |

+--------------+-------+-------+--------------------------------------+---------+------------+

root@controller01:/var/log/ceilometer# ceilometer meter-list
+------+------+------+-------------+---------+------------+
| Name | Type | Unit | Resource ID | User ID | Project ID |
+------+------+------+-------------+---------+------------+
+------+------+------+-------------+---------+------------+

There are no errors output, and it seems the command completes correctly.

So where am I going wrong here? Is there a step I am missing? Or is
there something obvious I have missed?

We have enabled glance, nova-compute, and cinder from the
instructions, but we aren't seeing any data (at least according to the
instructions).

Tom Walsh
Expresshosting.net


Mailing list:
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe :
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
responded Mar 2, 2016 by expresswebsys_at_gma (340 points)   1 1
0 votes

Tom,

From results you provided is seen that neither polling nor notification
data have consumers. So, first of all, check the status of
agent-notification service, something is wrong with it. Before it is fixed,
no sense to check Mongo. It just cannot contain any data.

Thanks,
Nadya

On Wed, Mar 2, 2016 at 6:48 PM, Tom Walsh expresswebsys@gmail.com wrote:

Nadya,

Okay. So looking at these two here is what I am seeing:

metering.sample 0 1

notifications.error 5 0
notifications.info 1139 0
notifications.sample 18488 0

So I am guessing that means we have a consumer for metering, but not a
consumer on notifications (and there is a bit of a backlog on those
queues). Is that a correct statement?

We are utilizing the mongodb backend for storage, but we aren't overly
familiar with it or how to inspect data within it. Any pointers for
checking that side of things?

On Wed, Mar 2, 2016 at 5:10 AM, Nadya Shakhat nprivalova@mirantis.com
wrote:

Hi Tom,

You may check several things:
1. Do you have messages in queues and do the queues have listeners? For
example, check collector-related queue 'metering.sample' by default.

rabbitmqctl list_queues name messages consumers | grep metering

rabbitmqctl list_queues name messages consumers | grep notification

The both queues should have consumers

  1. Check entries directly in db you use.

Thanks,

Nadya

On Wed, Mar 2, 2016 at 6:57 AM, Tom Walsh expresswebsys@gmail.com
wrote:

We currently have a working Liberty based Openstack deployment based
on Ubuntu LT 14.04 (2 controllers, 4 compute).

We have followed the instructions at

http://docs.openstack.org/liberty/install-guide-ubuntu/ceilometer-install.html

and everything seems to have installed correctly. No errors in the
logs, and nothing really obviously wrong.

However in the instructions there is this page

http://docs.openstack.org/liberty/install-guide-ubuntu/ceilometer-verify.html

that shows the output of meter-list, however our meter-list output is
blank.

Example output:

$ ceilometer meter-list

+--------------+-------+-------+--------------------------------------+---------+------------+

| Name | Type | Unit | Resource ID
| User ID | Project ID |

+--------------+-------+-------+--------------------------------------+---------+------------+

| image | gauge | image | acafc7c0-40aa-4026-9673-b879898e1fc2
| None | cf12a15... |
| image.size | gauge | B | acafc7c0-40aa-4026-9673-b879898e1fc2
| None | cf12a15... |

+--------------+-------+-------+--------------------------------------+---------+------------+

root@controller01:/var/log/ceilometer# ceilometer meter-list
+------+------+------+-------------+---------+------------+
| Name | Type | Unit | Resource ID | User ID | Project ID |
+------+------+------+-------------+---------+------------+
+------+------+------+-------------+---------+------------+

There are no errors output, and it seems the command completes
correctly.

So where am I going wrong here? Is there a step I am missing? Or is
there something obvious I have missed?

We have enabled glance, nova-compute, and cinder from the
instructions, but we aren't seeing any data (at least according to the
instructions).

Tom Walsh
Expresshosting.net


Mailing list:
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe :
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
responded Mar 2, 2016 by Nadya_Privalova (1,640 points)   1 3
0 votes

Nadya,

The logs for agent-notification appear to be clear of errors. Here is
an an excerpt from it: (I turned on debug to see if there is any
additional information that might be suppressed, but there doesn't
seem to be much there)

2016-03-02 13:43:23.194 31681 DEBUG ceilometer.pipeline [-] Pipeline
config file: /etc/ceilometer/pipeline.yaml setuppipelinemanager
/usr/lib/python2.7/dist-packages/ceilometer/pipeline.py:792
2016-03-02 13:43:23.209 31681 INFO ceilometer.pipeline [-] Pipeline
config: {'sources': [{'interval': 600, 'meters': ['*'], 'name':
'meter
source', 'sinks': ['metersink']}, {'interval': 600, 'meters':
['cpu'], 'name': 'cpu
source', 'sinks': ['cpusink',
'cpu
deltasink']}, {'interval': 600, 'meters': ['disk.read.bytes',
'disk.read.requests', 'disk.write.bytes', 'disk.write.requests',
'disk.device.read.bytes', 'disk.device.read.requests',
'disk.device.write.bytes', 'disk.device.write.requests'], 'name':
'disk
source', 'sinks': ['disksink']}, {'interval': 600, 'meters':
['network.incoming.bytes', 'network.incoming.packets',
'network.outgoing.bytes', 'network.outgoing.packets'], 'name':
'network
source', 'sinks': ['networksink']}], 'sinks':
[{'publishers': ['notifier://'], 'transformers': None, 'name':
'meter
sink'}, {'publishers': ['notifier://'], 'transformers':
[{'name': 'rateofchange', 'parameters': {'target': {'scale': '100.0
/ (109 * (resourcemetadata.cpunumber or 1))', 'type': 'gauge',
'name': 'cpuutil', 'unit': '%'}}}], 'name': 'cpusink'},
{'publishers': ['notifier://'], 'transformers': [{'name': 'delta',
'parameters': {'target': {'name': 'cpu.delta'}, 'growthonly':
True}}], 'name': 'cpu
deltasink'}, {'publishers': ['notifier://'],
'transformers': [{'name': 'rate
ofchange', 'parameters': {'source':
{'map
from': {'name':
'(disk\.device|disk)\.(read|write)\.(bytes|requests)', 'unit':
'(B|request)'}}, 'target': {'type': 'gauge', 'mapto': {'name':
'\1.\2.\3.rate', 'unit': '\1/s'}}}}], 'name': 'disk
sink'},
{'publishers': ['notifier://'], 'transformers': [{'name':
'rateofchange', 'parameters': {'source': {'mapfrom': {'name':
'network\.(incoming|outgoing)\.(bytes|packets)', 'unit':
'(B|packet)'}}, 'target': {'type': 'gauge', 'map
to': {'name':
'network.\1.\2.rate', 'unit': '\1/s'}}}}], 'name':
'network_sink'}]}
2016-03-02 13:43:23.262 31681 INFO ceilometer.pipeline [-] detected
decoupled pipeline config format
2016-03-02 13:43:23.263 31681 INFO ceilometer.publisher.messaging [-]
Publishing policy set to default
2016-03-02 13:43:23.283 31681 INFO ceilometer.publisher.messaging [-]
Publishing policy set to default
2016-03-02 13:43:23.283 31681 DEBUG ceilometer.transformer.conversions
[-] scaling conversion transformer with source: {} target: {'scale':
'100.0 / (10
9 * (resource_metadata.cpu_number or 1))', 'type':
'gauge', 'name': 'cpu_util', 'unit': '%'}: init
/usr/lib/python2.7/dist-packages/ceilometer/transformer/conversions.py:136
2016-03-02 13:43:23.292 31681 INFO ceilometer.pipeline [-] Pipeline
cpu_sink: Setup transformer instance rate_of_change with parameter
{'target': {'scale': '100.0 / (10**9 * (resource_metadata.cpu_number
or 1))', 'type': 'gauge', 'name': 'cpu_util', 'unit': '%'}}
2016-03-02 13:43:23.292 31681 INFO ceilometer.publisher.messaging [-]
Publishing policy set to default
2016-03-02 13:43:23.300 31681 INFO ceilometer.pipeline [-] Pipeline
cpu_delta_sink: Setup transformer instance delta with parameter
{'target': {'name': 'cpu.delta'}, 'growth_only': True}
2016-03-02 13:43:23.301 31681 INFO ceilometer.publisher.messaging [-]
Publishing policy set to default
2016-03-02 13:43:23.301 31681 DEBUG ceilometer.transformer.conversions
[-] scaling conversion transformer with source: {'map_from': {'name':
'(disk\.device|disk)\.(read|write)\.(bytes|requests)', 'unit':
'(B|request)'}} target: {'map_to': {'name': '\1.\2.\3.rate',
'unit': '\1/s'}, 'type': 'gauge'}: __init__
/usr/lib/python2.7/dist-packages/ceilometer/transformer/conversions.py:136
2016-03-02 13:43:23.310 31681 INFO ceilometer.pipeline [-] Pipeline
disk_sink: Setup transformer instance rate_of_change with parameter
{'source': {'map_from': {'name':
'(disk\.device|disk)\.(read|write)\.(bytes|requests)', 'unit':
'(B|request)'}}, 'target': {'type': 'gauge', 'map_to': {'name':
'\1.\2.\3.rate', 'unit': '\1/s'}}}
2016-03-02 13:43:23.311 31681 INFO ceilometer.publisher.messaging [-]
Publishing policy set to default
2016-03-02 13:43:23.311 31681 DEBUG ceilometer.transformer.conversions
[-] scaling conversion transformer with source: {'map_from': {'name':
'network\.(incoming|outgoing)\.(bytes|packets)', 'unit':
'(B|packet)'}} target: {'map_to': {'name': 'network.\1.\2.rate',
'unit': '\1/s'}, 'type': 'gauge'}: __init__
/usr/lib/python2.7/dist-packages/ceilometer/transformer/conversions.py:136
2016-03-02 13:43:23.319 31681 INFO ceilometer.pipeline [-] Pipeline
network_sink: Setup transformer instance rate_of_change with parameter
{'source': {'map_from': {'name':
'network\.(incoming|outgoing)\.(bytes|packets)', 'unit':
'(B|packet)'}}, 'target': {'type': 'gauge', 'map_to': {'name':
'network.\1.\2.rate', 'unit': '\1/s'}}}
2016-03-02 13:43:23.329 31681 DEBUG ceilometer.meter.notifications [-]
Meter Definitions configuration file:
/usr/lib/python2.7/dist-packages/ceilometer/meter/data/meters.yaml
setup_meters_config
/usr/lib/python2.7/dist-packages/ceilometer/meter/notifications.py:150
2016-03-02 13:43:23.447 31681 INFO ceilometer.meter.notifications [-]
Meter Definitions: {'metric': [{'name': 'image.size', 'resource_id':
'$.payload.id', 'volume': '$.payload.size', 'project_id':
'$.payload.owner', 'type': 'gauge', 'unit': 'B', 'event_type':
['image.upload', 'image.delete', 'image.update']}, {'user_id':
'$.payload.receiver_user_id', 'name': 'image.download', 'resource_id':
'$.payload.image_id', 'volume': '$.payload.bytes_sent', 'project_id':
'$.payload.receiver_tenant_id', 'type': 'delta', 'unit': 'B',
'event_type': 'image.send'}, {'name': 'image.serve', 'resource_id':
'$.payload.image_id', 'volume': '$.payload.bytes_sent', 'project_id':
'$.payload.owner_id', 'type': 'delta', 'unit': 'B', 'event_type':
'image.send'}, {'user_id': '$._context_user', 'name':
'magnetodb.table.index.count', 'resource_id': '$.payload.table_uuid',
'volume': '$.payload.index_count', 'type': 'gauge', 'unit': 'index',
'event_type': 'magnetodb.table.create.end'}, {'user_id':
'$.payload.user_id', 'name': 'volume.size', 'resource_id':
'$.payload.volume_id', 'volume': '$.payload.size', 'project_id':
'$.payload.tenant_id', 'type': 'gauge', 'unit': 'GB', 'event_type':
['volume.exists', 'volume.create.*', 'volume.delete.*',
'volume.resize.*', 'volume.attach.*', 'volume.detach.*',
'volume.update.*']}, {'user_id': '$.payload.user_id', 'name':
'snapshot.size', 'resource_id': '$.payload.snapshot_id', 'volume':
'$.payload.volume_size', 'project_id': '$.payload.tenant_id', 'type':
'gauge', 'unit': 'GB', 'event_type': ['snapshot.exists',
'snapshot.create.*', 'snapshot.delete.*']}, {'user_id':
'$.payload.user_id', 'name': '$.payload.metrics.[*].name',
'resource_id': '$.payload.resource_id', 'volume':
'$.payload.metrics.[*].value', 'lookup': ['name', 'unit', 'volume'],
'project_id': '$.payload.project_id', 'type': 'gauge', 'unit':
'$.payload.metrics.[*].unit', 'event_type': 'magnum.bay.metrics.*'},
{'user_id': '$.payload.initiator.id', 'name':
'$.payload.measurements.[*].metric.[*].name', 'resource_id':
'$.payload.target.id', 'volume': '$.payload.measurements.[*].result',
'lookup': ['name', 'unit', 'volume'], 'project_id':
'$.payload.initiator.project_id', 'type': 'delta', 'unit':
'$.payload.measurements.[*].metric.[*].unit', 'event_type':
'objectstore.http.request'}, {'user_id': '$.payload.user_id', 'name':
'memory', 'resource_id': '$.payload.instance_id', 'volume':
'$.payload.memory_mb', 'project_id': '$.payload.tenant_id', 'type':
'gauge', 'unit': 'MB', 'event_type': 'compute.instance.*'},
{'user_id': '$.payload.user_id', 'name': 'vcpus', 'resource_id':
'$.payload.instance_id', 'volume': '$.payload.vcpus', 'project_id':
'$.payload.tenant_id', 'type': 'gauge', 'unit': 'vcpu', 'event_type':
'compute.instance.*'}, {'user_id': '$.payload.user_id', 'name':
'disk.root.size', 'resource_id': '$.payload.instance_id', 'volume':
'$.payload.root_gb', 'project_id': '$.payload.tenant_id', 'type':
'gauge', 'unit': 'GB', 'event_type': 'compute.instance.*'},
{'user_id': '$.payload.user_id', 'name': 'disk.ephemeral.size',
'resource_id': '$.payload.instance_id', 'volume':
'$.payload.ephemeral_gb', 'project_id': '$.payload.tenant_id', 'type':
'gauge', 'unit': 'GB', 'event_type': 'compute.instance.*'}, {'name':
'bandwidth', 'resource_id': '$.payload.label_id', 'volume':
'$.payload.bytes', 'project_id': '$.payload.tenant_id', 'type':
'delta', 'unit': 'B', 'event_type': 'l3.meter'}, {'name':
'compute.node.cpu.frequency', 'resource_id': '$.payload.host + "_" +
$.payload.nodename', 'timestamp':
"$.payload.metrics[?(@.name='cpu.frequency')].timestamp", 'volume':
"$.payload.metrics[?(@.name='cpu.frequency')].value", 'metadata':
{'source': "$.payload.metrics[?(@.name='cpu.frequency')].source",
'host': '$.publisher_id', 'event_type': '$.event_type'}, 'type':
'gauge', 'unit': 'MHz', 'event_type': 'compute.metrics.update'},
{'name': 'compute.node.cpu.user.time', 'resource_id': '$.payload.host
+ "_" + $.payload.nodename', 'timestamp':
"$.payload.metrics[?(@.name='cpu.user.time')].timestamp", 'volume':
"$.payload.metrics[?(@.name='cpu.user.time')].value", 'metadata':
{'source': "$.payload.metrics[?(@.name='cpu.user.time')].source",
'host': '$.publisher_id', 'event_type': '$.event_type'}, 'type':
'cumulative', 'unit': 'ns', 'event_type': 'compute.metrics.update'},
{'name': 'compute.node.cpu.kernel.time', 'resource_id':
'$.payload.host + "_" + $.payload.nodename', 'timestamp':
"$.payload.metrics[?(@.name='cpu.kernel.time')].timestamp", 'volume':
"$.payload.metrics[?(@.name='cpu.kernel.time')].value", 'metadata':
{'source': "$.payload.metrics[?(@.name='cpu.kernel.time')].source",
'host': '$.publisher_id', 'event_type': '$.event_type'}, 'type':
'cumulative', 'unit': 'ns', 'event_type': 'compute.metrics.update'},
{'name': 'compute.node.cpu.idle.time', 'resource_id': '$.payload.host
+ "_" + $.payload.nodename', 'timestamp':
"$.payload.metrics[?(@.name='cpu.idle.time')].timestamp", 'volume':
"$.payload.metrics[?(@.name='cpu.idle.time')].value", 'metadata':
{'source': "$.payload.metrics[?(@.name='cpu.idle.time')].source",
'host': '$.publisher_id', 'event_type': '$.event_type'}, 'type':
'cumulative', 'unit': 'ns', 'event_type': 'compute.metrics.update'},
{'name': 'compute.node.cpu.iowait.time', 'resource_id':
'$.payload.host + "_" + $.payload.nodename', 'timestamp':
"$.payload.metrics[?(@.name='cpu.iowait.time')].timestamp", 'volume':
"$.payload.metrics[?(@.name='cpu.iowait.time')].value", 'metadata':
{'source': "$.payload.metrics[?(@.name='cpu.iowait.time')].source",
'host': '$.publisher_id', 'event_type': '$.event_type'}, 'type':
'cumulative', 'unit': 'ns', 'event_type': 'compute.metrics.update'},
{'name': 'compute.node.cpu.kernel.percent', 'resource_id':
'$.payload.host + "_" + $.payload.nodename', 'timestamp':
"$.payload.metrics[?(@.name='cpu.kernel.percent')].timestamp",
'volume': "$.payload.metrics[?(@.name='cpu.kernel.percent')].value *
100", 'metadata': {'source':
"$.payload.metrics[?(@.name='cpu.kernel.percent')].source", 'host':
'$.publisher_id', 'event_type': '$.event_type'}, 'type': 'gauge',
'unit': 'percent', 'event_type': 'compute.metrics.update'}, {'name':
'compute.node.cpu.idle.percent', 'resource_id': '$.payload.host + "_"
+ $.payload.nodename', 'timestamp':
"$.payload.metrics[?(@.name='cpu.idle.percent')].timestamp", 'volume':
"$.payload.metrics[?(@.name='cpu.idle.percent')].value * 100",
'metadata': {'source':
"$.payload.metrics[?(@.name='cpu.idle.percent')].source", 'host':
'$.publisher_id', 'event_type': '$.event_type'}, 'type': 'gauge',
'unit': 'percent', 'event_type': 'compute.metrics.update'}, {'name':
'compute.node.cpu.user.percent', 'resource_id': '$.payload.host + "_"
+ $.payload.nodename', 'timestamp':
"$.payload.metrics[?(@.name='cpu.user.percent')].timestamp", 'volume':
"$.payload.metrics[?(@.name='cpu.user.percent')].value * 100",
'metadata': {'source':
"$.payload.metrics[?(@.name='cpu.user.percent')].source", 'host':
'$.publisher_id', 'event_type': '$.event_type'}, 'type': 'gauge',
'unit': 'percent', 'event_type': 'compute.metrics.update'}, {'name':
'compute.node.cpu.iowait.percent', 'resource_id': '$.payload.host +
"_" + $.payload.nodename', 'timestamp':
"$.payload.metrics[?(@.name='cpu.iowait.percent')].timestamp",
'volume': "$.payload.metrics[?(@.name='cpu.iowait.percent')].value *
100", 'metadata': {'source':
"$.payload.metrics[?(@.name='cpu.iowait.percent')].source", 'host':
'$.publisher_id', 'event_type': '$.event_type'}, 'type': 'gauge',
'unit': 'percent', 'event_type': 'compute.metrics.update'}, {'name':
'compute.node.cpu.percent', 'resource_id': '$.payload.host + "_" +
$.payload.nodename', 'timestamp':
"$.payload.metrics[?(@.name='cpu.percent')].timestamp", 'volume':
"$.payload.metrics[?(@.name='cpu.percent')].value * 100", 'metadata':
{'source': "$.payload.metrics[?(@.name='cpu.percent')].source",
'host': '$.publisher_id', 'event_type': '$.event_type'}, 'type':
'gauge', 'unit': 'percent', 'event_type': 'compute.metrics.update'},
{'name': 'image', 'resource_id': '$.payload.id', 'volume': 1,
'project_id': '$.payload.owner', 'type': 'gauge', 'unit': 'image',
'event_type': ['image.upload', 'image.update', 'image.delete']},
{'name': 'image.upload', 'resource_id': '$.payload.id', 'volume': 1,
'project_id': '$.payload.owner', 'type': 'gauge', 'unit': 'image',
'event_type': ['image.upload']}, {'name': 'image.delete',
'resource_id': '$.payload.id', 'volume': 1, 'project_id':
'$.payload.owner', 'type': 'gauge', 'unit': 'image', 'event_type':
['image.delete']}, {'name': 'image.update', 'resource_id':
'$.payload.id', 'volume': 1, 'project_id': '$.payload.owner', 'type':
'gauge', 'unit': 'image', 'event_type': ['image.update']}, {'user_id':
'_context_trustor_user_id', 'name': 'stack.create', 'resource_id':
'$.payload.stack_identity', 'volume': 1, 'project_id':
'$.payload.tenant_id', 'type': 'delta', 'unit': 'stack', 'event_type':
['orchestration.stack.create.end']}, {'user_id':
'_context_trustor_user_id', 'name': 'stack.update', 'resource_id':
'$.payload.stack_identity', 'volume': 1, 'project_id':
'$.payload.tenant_id', 'type': 'delta', 'unit': 'stack', 'event_type':
['orchestration.stack.update.end']}, {'user_id':
'_context_trustor_user_id', 'name': 'stack.delete', 'resource_id':
'$.payload.stack_identity', 'volume': 1, 'project_id':
'$.payload.tenant_id', 'type': 'delta', 'unit': 'stack', 'event_type':
['orchestration.stack.delete.end']}, {'user_id':
'_context_trustor_user_id', 'name': 'stack.resume', 'resource_id':
'$.payload.stack_identity', 'volume': 1, 'project_id':
'$.payload.tenant_id', 'type': 'delta', 'unit': 'stack', 'event_type':
['orchestration.stack.resume.end']}, {'user_id':
'_context_trustor_user_id', 'name': 'stack.suspend', 'resource_id':
'$.payload.stack_identity', 'volume': 1, 'project_id':
'$.payload.tenant_id', 'type': 'delta', 'unit': 'stack', 'event_type':
['orchestration.stack.suspend.end']}, {'user_id': '_context_user',
'name': 'magnetodb.table.create', 'resource_id':
'$.payload.table_uuid', 'volume': 1, 'project_id': '_context_tenant',
'type': 'gauge', 'unit': 'table', 'event_type':
'magnetodb.table.create.end'}, {'user_id': '_context_user', 'name':
'magnetodb.table.delete', 'resource_id': '$.payload.table_uuid',
'volume': 1, 'project_id': '_context_tenant', 'type': 'gauge', 'unit':
'table', 'event_type': 'magnetodb.table.delete.end'}, {'user_id':
'$.payload.user_id', 'name': 'volume', 'resource_id':
'$.payload.volume_id', 'volume': 1, 'project_id':
'$.payload.tenant_id', 'type': 'gauge', 'unit': 'volume',
'event_type': ['volume.exists', 'volume.create.*', 'volume.delete.*',
'volume.resize.*', 'volume.attach.*', 'volume.detach.*',
'volume.update.*']}, {'user_id': '$.payload.user_id', 'name':
'volume.exists', 'resource_id': '$.payload.volume_id', 'volume': 1,
'project_id': '$.payload.tenant_id', 'type': 'delta', 'unit':
'volume', 'event_type': ['volume.exists']}, {'user_id':
'$.payload.user_id', 'name': 'volume.create.start', 'resource_id':
'$.payload.volume_id', 'volume': 1, 'project_id':
'$.payload.tenant_id', 'type': 'delta', 'unit': 'volume',
'event_type': ['volume.create.start']}, {'user_id':
'$.payload.user_id', 'name': 'volume.create.end', 'resource_id':
'$.payload.volume_id', 'volume': 1, 'project_id':
'$.payload.tenant_id', 'type': 'delta', 'unit': 'volume',
'event_type': ['volume.create.end']}, {'user_id': '$.payload.user_id',
'name': 'volume.delete.start', 'resource_id': '$.payload.volume_id',
'volume': 1, 'project_id': '$.payload.tenant_id', 'type': 'delta',
'unit': 'volume', 'event_type': ['volume.delete.start']}, {'user_id':
'$.payload.user_id', 'name': 'volume.delete.end', 'resource_id':
'$.payload.volume_id', 'volume': 1, 'project_id':
'$.payload.tenant_id', 'type': 'delta', 'unit': 'volume',
'event_type': ['volume.delete.end']}, {'user_id': '$.payload.user_id',
'name': 'volume.update.end', 'resource_id': '$.payload.volume_id',
'volume': 1, 'project_id': '$.payload.tenant_id', 'type': 'delta',
'unit': 'volume', 'event_type': ['volume.update.end']}, {'user_id':
'$.payload.user_id', 'name': 'volume.update.start', 'resource_id':
'$.payload.volume_id', 'volume': 1, 'project_id':
'$.payload.tenant_id', 'type': 'delta', 'unit': 'volume',
'event_type': ['volume.update.start']}, {'user_id':
'$.payload.user_id', 'name': 'volume.resize.end', 'resource_id':
'$.payload.volume_id', 'volume': 1, 'project_id':
'$.payload.tenant_id', 'type': 'delta', 'unit': 'volume',
'event_type': ['volume.resize.end']}, {'user_id': '$.payload.user_id',
'name': 'volume.resize.start', 'resource_id': '$.payload.volume_id',
'volume': 1, 'project_id': '$.payload.tenant_id', 'type': 'delta',
'unit': 'volume', 'event_type': ['volume.resize.start']}, {'user_id':
'$.payload.user_id', 'name': 'volume.attach.end', 'resource_id':
'$.payload.volume_id', 'volume': 1, 'project_id':
'$.payload.tenant_id', 'type': 'delta', 'unit': 'volume',
'event_type': ['volume.attach.end']}, {'user_id': '$.payload.user_id',
'name': 'volume.attach.start', 'resource_id': '$.payload.volume_id',
'volume': 1, 'project_id': '$.payload.tenant_id', 'type': 'delta',
'unit': 'volume', 'event_type': ['volume.attach.start']}, {'user_id':
'$.payload.user_id', 'name': 'volume.detach.end', 'resource_id':
'$.payload.volume_id', 'volume': 1, 'project_id':
'$.payload.tenant_id', 'type': 'delta', 'unit': 'volume',
'event_type': ['volume.detach.end']}, {'user_id': '$.payload.user_id',
'name': 'volume.detach.start', 'resource_id': '$.payload.volume_id',
'volume': 1, 'project_id': '$.payload.tenant_id', 'type': 'delta',
'unit': 'volume', 'event_type': ['volume.detach.start']}, {'user_id':
'$.payload.user_id', 'name': 'snapshot', 'resource_id':
'$.payload.snapshot_id', 'volume': 1, 'project_id':
'$.payload.tenant_id', 'type': 'gauge', 'unit': 'snapshot',
'event_type': ['snapshot.exists', 'snapshot.create.*',
'snapshot.delete.*']}, {'user_id': '$.payload.user_id', 'name':
'snapshot.exists', 'resource_id': '$.payload.snapshot_id', 'volume':
1, 'project_id': '$.payload.tenant_id', 'type': 'delta', 'unit':
'snapshot', 'event_type': ['snapshot.exists']}, {'user_id':
'$.payload.user_id', 'name': 'snapshot.create.start', 'resource_id':
'$.payload.snapshot_id', 'volume': 1, 'project_id':
'$.payload.tenant_id', 'type': 'delta', 'unit': 'snapshot',
'event_type': ['snapshot.create.start']}, {'user_id':
'$.payload.user_id', 'name': 'snapshot.create.end', 'resource_id':
'$.payload.snapshot_id', 'volume': 1, 'project_id':
'$.payload.tenant_id', 'type': 'delta', 'unit': 'snapshot',
'event_type': ['snapshot.create.end']}, {'user_id':
'$.payload.user_id', 'name': 'snapshot.delete.start', 'resource_id':
'$.payload.snapshot_id', 'volume': 1, 'project_id':
'$.payload.tenant_id', 'type': 'delta', 'unit': 'snapshot',
'event_type': ['snapshot.delete.start']}, {'user_id':
'$.payload.user_id', 'name': 'snapshot.delete.end', 'resource_id':
'$.payload.snapshot_id', 'volume': 1, 'project_id':
'$.payload.tenant_id', 'type': 'delta', 'unit': 'snapshot',
'event_type': ['snapshot.delete.end']}, {'name': 'cluster.create',
'resource_id': '$.payload.cluster_id', 'volume': 1, 'project_id':
'$.payload.project_id', 'type': 'delta', 'unit': 'cluster',
'event_type': ['sahara.cluster.create']}, {'name': 'cluster.update',
'resource_id': '$.payload.cluster_id', 'volume': 1, 'project_id':
'$.payload.project_id', 'type': 'delta', 'unit': 'cluster',
'event_type': ['sahara.cluster.update']}, {'name': 'cluster.delete',
'resource_id': '$.payload.cluster_id', 'volume': 1, 'project_id':
'$.payload.project_id', 'type': 'delta', 'unit': 'cluster',
'event_type': ['sahara.cluster.delete']}, {'user_id':
'$.payload.initiator.id', 'name': 'identity.user.created',
'resource_id': '$.payload.resource_info', 'volume': 1, 'type':
'delta', 'unit': 'user', 'event_type': ['identity.user.created']},
{'user_id': '$.payload.initiator.id', 'name': 'identity.user.updated',
'resource_id': '$.payload.resource_info', 'volume': 1, 'type':
'delta', 'unit': 'user', 'event_type': ['identity.user.updated']},
{'user_id': '$.payload.initiator.id', 'name': 'identity.user.deleted',
'resource_id': '$.payload.resource_info', 'volume': 1, 'type':
'delta', 'unit': 'user', 'event_type': ['identity.user.deleted']},
{'user_id': '$.payload.initiator.id', 'name':
'identity.group.created', 'resource_id': '$.payload.resource_info',
'volume': 1, 'type': 'delta', 'unit': 'group', 'event_type':
['identity.group.created']}, {'user_id': '$.payload.initiator.id',
'name': 'identity.group.updated', 'resource_id':
'$.payload.resource_info', 'volume': 1, 'type': 'delta', 'unit':
'group', 'event_type': ['identity.group.updated']}, {'user_id':
'$.payload.initiator.id', 'name': 'identity.group.deleted',
'resource_id': '$.payload.resource_info', 'volume': 1, 'type':
'delta', 'unit': 'group', 'event_type': ['identity.group.deleted']},
{'user_id': '$.payload.initiator.id', 'name':
'identity.project.created', 'resource_id': '$.payload.resource_info',
'volume': 1, 'type': 'delta', 'unit': 'project', 'event_type':
['identity.project.created']}, {'user_id': '$.payload.initiator.id',
'name': 'identity.project.updated', 'resource_id':
'$.payload.resource_info', 'volume': 1, 'type': 'delta', 'unit':
'project', 'event_type': ['identity.project.updated']}, {'user_id':
'$.payload.initiator.id', 'name': 'identity.project.deleted',
'resource_id': '$.payload.resource_info', 'volume': 1, 'type':
'delta', 'unit': 'project', 'event_type':
['identity.project.deleted']}, {'user_id': '$.payload.initiator.id',
'name': 'identity.role.created', 'resource_id':
'$.payload.resource_info', 'volume': 1, 'type': 'delta', 'unit':
'role', 'event_type': ['identity.role.created']}, {'user_id':
'$.payload.initiator.id', 'name': 'identity.role.updated',
'resource_id': '$.payload.resource_info', 'volume': 1, 'type':
'delta', 'unit': 'role', 'event_type': ['identity.role.updated']},
{'user_id': '$.payload.initiator.id', 'name': 'identity.role.deleted',
'resource_id': '$.payload.resource_info', 'volume': 1, 'type':
'delta', 'unit': 'role', 'event_type': ['identity.role.deleted']},
{'user_id': '$.payload.initiator.id', 'name':
'identity.role_assignment.created', 'resource_id': '$.payload.role',
'volume': 1, 'type': 'delta', 'unit': 'role_assignment', 'event_type':
['identity.role_assignment.created']}, {'user_id':
'$.payload.initiator.id', 'name': 'identity.role_assignment.deleted',
'resource_id': '$.payload.role', 'volume': 1, 'type': 'delta', 'unit':
'role_assignment', 'event_type':
['identity.role_assignment.deleted']}, {'user_id':
'$.payload.initiator.id', 'name': 'identity.authenticate.success',
'resource_id': '$.payload.initiator.id', 'volume': 1, 'type': 'delta',
'unit': 'user', 'event_type': ['identity.authenticate']}, {'user_id':
'$.payload.initiator.id', 'name': 'identity.authenticate.pending',
'resource_id': '$.payload.initiator.id', 'volume': 1, 'type': 'delta',
'unit': 'user', 'event_type': ['identity.authenticate']}, {'user_id':
'$.payload.initiator.id', 'name': 'identity.authenticate.failure',
'resource_id': '$.payload.initiator.id', 'volume': 1, 'type': 'delta',
'unit': 'user', 'event_type': ['identity.authenticate']}, {'user_id':
'$.payload.initiator.id', 'name': 'identity.trust.created',
'resource_id': '$.payload.resource_info', 'volume': 1, 'type':
'delta', 'unit': 'trust', 'event_type':
['identity.OS-TRUST:trust.created']}, {'user_id':
'$.payload.initiator.id', 'name': 'identity.trust.deleted',
'resource_id': '$.payload.resource_info', 'volume': 1, 'type':
'delta', 'unit': 'trust', 'event_type':
['identity.OS-TRUST:trust.deleted']}, {'user_id':
'$.payload.initiator.id', 'name': 'storage.api.request',
'resource_id': '$.payload.target.id', 'volume': 1, 'project_id':
'$.payload.initiator.project_id', 'type': 'delta', 'unit': 'request',
'event_type': ['objectstore.http.request']}, {'user_id':
'$.payload.user_id', 'name': '$.payload.name', 'resource_id':
'"profiler-" + $.payload.base_id', 'volume': 1, 'project_id':
'$.payload.project_id', 'type': 'gauge', 'unit': 'trace',
'event_type': 'profiler.*'}]}

So there are no exceptions being thrown... the process seems to be
running correctly.

Tom Walsh
ExpressHosting.net

On Wed, Mar 2, 2016 at 12:19 PM, Nadya Shakhat nprivalova@mirantis.com wrote:
Tom,

From results you provided is seen that neither polling nor notification data
have consumers. So, first of all, check the status of agent-notification
service, something is wrong with it. Before it is fixed, no sense to check
Mongo. It just cannot contain any data.

Thanks,
Nadya

On Wed, Mar 2, 2016 at 6:48 PM, Tom Walsh expresswebsys@gmail.com wrote:

Nadya,

Okay. So looking at these two here is what I am seeing:

metering.sample 0 1

notifications.error 5 0
notifications.info 1139 0
notifications.sample 18488 0

So I am guessing that means we have a consumer for metering, but not a
consumer on notifications (and there is a bit of a backlog on those
queues). Is that a correct statement?

We are utilizing the mongodb backend for storage, but we aren't overly
familiar with it or how to inspect data within it. Any pointers for
checking that side of things?

On Wed, Mar 2, 2016 at 5:10 AM, Nadya Shakhat nprivalova@mirantis.com
wrote:

Hi Tom,

You may check several things:
1. Do you have messages in queues and do the queues have listeners? For
example, check collector-related queue 'metering.sample' by default.

rabbitmqctl list_queues name messages consumers | grep metering

rabbitmqctl list_queues name messages consumers | grep notification

The both queues should have consumers

  1. Check entries directly in db you use.

Thanks,

Nadya

On Wed, Mar 2, 2016 at 6:57 AM, Tom Walsh expresswebsys@gmail.com
wrote:

We currently have a working Liberty based Openstack deployment based
on Ubuntu LT 14.04 (2 controllers, 4 compute).

We have followed the instructions at

http://docs.openstack.org/liberty/install-guide-ubuntu/ceilometer-install.html
and everything seems to have installed correctly. No errors in the
logs, and nothing really obviously wrong.

However in the instructions there is this page

http://docs.openstack.org/liberty/install-guide-ubuntu/ceilometer-verify.html
that shows the output of meter-list, however our meter-list output is
blank.

Example output:

$ ceilometer meter-list

+--------------+-------+-------+--------------------------------------+---------+------------+
| Name | Type | Unit | Resource ID
| User ID | Project ID |

+--------------+-------+-------+--------------------------------------+---------+------------+
| image | gauge | image | acafc7c0-40aa-4026-9673-b879898e1fc2
| None | cf12a15... |
| image.size | gauge | B | acafc7c0-40aa-4026-9673-b879898e1fc2
| None | cf12a15... |

+--------------+-------+-------+--------------------------------------+---------+------------+

root@controller01:/var/log/ceilometer# ceilometer meter-list
+------+------+------+-------------+---------+------------+
| Name | Type | Unit | Resource ID | User ID | Project ID |
+------+------+------+-------------+---------+------------+
+------+------+------+-------------+---------+------------+

There are no errors output, and it seems the command completes
correctly.

So where am I going wrong here? Is there a step I am missing? Or is
there something obvious I have missed?

We have enabled glance, nova-compute, and cinder from the
instructions, but we aren't seeing any data (at least according to the
instructions).

Tom Walsh
Expresshosting.net


Mailing list:
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe :
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
responded Mar 2, 2016 by expresswebsys_at_gma (340 points)   1 1
0 votes

In case anybody runs into this issue in the future, we have determined
what the issue is and how to correct it.

First the ceilometer-agent-notification process seems to be
suppressing the exceptions that were being thrown from the daemon.
This is why the logs didn't seem to shine any light on the issue. The
logs were missing a critical exception message. So I had to run the
agent manually with the --verbose switch:

/usr/bin/python /usr/bin/ceilometer-agent-notification --verbose
--config-file=/etc/ceilometer/ceilometer.conf

Which then yielded this exception:

Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py",
line 457, in fire_timers
timer()
File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/timer.py",
line 58, in call
cb(*args, **kw)
File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py",
line 214, in main
result = function(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/oslo_service/service.py",
line 671, in run_service
service.start()
File "/usr/lib/python2.7/dist-packages/ceilometer/notification.py",
line 174, in start
self.event_pipe_manager)
File "/usr/lib/python2.7/dist-packages/ceilometer/notification.py",
line 223, in _configure_main_queue_listeners
for new_tar in handler.get_targets(cfg.CONF):
File "/usr/lib/python2.7/dist-packages/ceilometer/ipmi/notifications/ironic.py",
line 87, in get_targets
for topic in conf.notification_topics]
File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py",
line 2062, in __getattr__
raise NoSuchOptError(name)
NoSuchOptError: no such option in group DEFAULT: notification_topics

A little quick googling resulted in this patchset:

https://review.openstack.org/#/c/255296/

After applying this patchset to the current ceilometer installation we
are seeing the notifications from the message bus being processed by
the ceilometer agent and then passed to the ceilometer collector.

My guess is that the underlying changes to oslo.messaging have now
broken the config parsing in ceilometer.

It appears that this patchset is missing from the current Ubuntu
cloudarchive ceilometer install (1:5.0.1-0ubuntu1~cloud0)

I hope that helps somebody... I spent enough time on it.

Tom Walsh
ExpressHosting.net


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
responded Mar 3, 2016 by expresswebsys_at_gma (340 points)   1 1
0 votes

Tom,

Thanks you very much for this! ceilometer.conf file contains "verbose"
option. Perhaps, switching it to True would have shown any errors too. I'll
check this behaviour one more time.

Thanks,
Nadya

On Thu, Mar 3, 2016 at 9:29 PM, Tom Walsh expresswebsys@gmail.com wrote:

In case anybody runs into this issue in the future, we have determined
what the issue is and how to correct it.

First the ceilometer-agent-notification process seems to be
suppressing the exceptions that were being thrown from the daemon.
This is why the logs didn't seem to shine any light on the issue. The
logs were missing a critical exception message. So I had to run the
agent manually with the --verbose switch:

/usr/bin/python /usr/bin/ceilometer-agent-notification --verbose
--config-file=/etc/ceilometer/ceilometer.conf

Which then yielded this exception:

Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py",
line 457, in fire_timers
timer()
File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/timer.py",
line 58, in call
cb(*args, **kw)
File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py",
line 214, in main
result = function(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/oslo_service/service.py",
line 671, in run_service
service.start()
File "/usr/lib/python2.7/dist-packages/ceilometer/notification.py",
line 174, in start
self.event_pipe_manager)
File "/usr/lib/python2.7/dist-packages/ceilometer/notification.py",
line 223, in _configure_main_queue_listeners
for new_tar in handler.get_targets(cfg.CONF):
File
"/usr/lib/python2.7/dist-packages/ceilometer/ipmi/notifications/ironic.py",
line 87, in get_targets
for topic in conf.notification_topics]
File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py",
line 2062, in __getattr__
raise NoSuchOptError(name)
NoSuchOptError: no such option in group DEFAULT: notification_topics

A little quick googling resulted in this patchset:

https://review.openstack.org/#/c/255296/

After applying this patchset to the current ceilometer installation we
are seeing the notifications from the message bus being processed by
the ceilometer agent and then passed to the ceilometer collector.

My guess is that the underlying changes to oslo.messaging have now
broken the config parsing in ceilometer.

It appears that this patchset is missing from the current Ubuntu
cloudarchive ceilometer install (1:5.0.1-0ubuntu1~cloud0)

I hope that helps somebody... I spent enough time on it.

Tom Walsh
ExpressHosting.net


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
responded Mar 4, 2016 by Nadya_Privalova (1,640 points)   1 3
0 votes

Nadya,

We had it set verbose = True in the conf file but it did not reveal this error.

On Fri, Mar 4, 2016 at 4:50 AM, Nadya Shakhat nprivalova@mirantis.com wrote:
Tom,

Thanks you very much for this! ceilometer.conf file contains "verbose"
option. Perhaps, switching it to True would have shown any errors too. I'll
check this behaviour one more time.

Thanks,
Nadya

On Thu, Mar 3, 2016 at 9:29 PM, Tom Walsh expresswebsys@gmail.com wrote:

In case anybody runs into this issue in the future, we have determined
what the issue is and how to correct it.

First the ceilometer-agent-notification process seems to be
suppressing the exceptions that were being thrown from the daemon.
This is why the logs didn't seem to shine any light on the issue. The
logs were missing a critical exception message. So I had to run the
agent manually with the --verbose switch:

/usr/bin/python /usr/bin/ceilometer-agent-notification --verbose
--config-file=/etc/ceilometer/ceilometer.conf

Which then yielded this exception:

Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py",
line 457, in fire_timers
timer()
File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/timer.py",
line 58, in call
cb(*args, **kw)
File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py",
line 214, in main
result = function(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/oslo_service/service.py",
line 671, in run_service
service.start()
File "/usr/lib/python2.7/dist-packages/ceilometer/notification.py",
line 174, in start
self.event_pipe_manager)
File "/usr/lib/python2.7/dist-packages/ceilometer/notification.py",
line 223, in _configure_main_queue_listeners
for new_tar in handler.get_targets(cfg.CONF):
File
"/usr/lib/python2.7/dist-packages/ceilometer/ipmi/notifications/ironic.py",
line 87, in get_targets
for topic in conf.notification_topics]
File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py",
line 2062, in __getattr__
raise NoSuchOptError(name)
NoSuchOptError: no such option in group DEFAULT: notification_topics

A little quick googling resulted in this patchset:

https://review.openstack.org/#/c/255296/

After applying this patchset to the current ceilometer installation we
are seeing the notifications from the message bus being processed by
the ceilometer agent and then passed to the ceilometer collector.

My guess is that the underlying changes to oslo.messaging have now
broken the config parsing in ceilometer.

It appears that this patchset is missing from the current Ubuntu
cloudarchive ceilometer install (1:5.0.1-0ubuntu1~cloud0)

I hope that helps somebody... I spent enough time on it.

Tom Walsh
ExpressHosting.net


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
responded Mar 5, 2016 by expresswebsys_at_gma (340 points)   1 1
0 votes

Hi list,

last week we upgraded our Mitaka cloud to Ocata (via Newton, of
course) with ceph backend, and also upgraded the cloud nodes from
openSUSE Leap 42.1 to Leap 42.3. There were some issues as expected,
but no showstoppers (luckily).
So the cloud is up and working again, but our monitoring shows a high
CPU load for cinder-volume service on the control node. But since all
the clients are on the compute nodes we are wondering what cinder
actually does on the control node except initializing the connections
of course. I captured a tcpdump on control node and saw a lot of
connections to the ceph nodes, the data contains all these rbd_header
files, e.g. rb.0.24d5b04[...]. I expect this kind of traffic on the
compute nodes, of course, but why does the control node also establish
so many connections?

I'd appreciate any insight!

Regards,
Eugen

--
Eugen Block voice : +49-40-559 51 75
NDE Netzdesign und -entwicklung AG fax : +49-40-559 51 77
Postfach 61 03 15
D-22423 Hamburg e-mail : eblock@nde.ag

     Vorsitzende des Aufsichtsrates: Angelika Mozdzen
       Sitz und Registergericht: Hamburg, HRB 90934
               Vorstand: Jens-U. Mozdzen
                USt-IdNr. DE 814 013 983
responded Oct 16, 2017 by Eugen_Block (3,740 points)   2 2
...