settingsLogin | Registersettings

[openstack-dev] Logging format: let's discuss a bit about default format, format configuration and so on

0 votes

Dear Stackers,

While working on my locally deployed Openstack (Pike using TripleO), I
was a bit struggling with the logging part. Currently, all logs are
pushed to per-service files, in the standard format "one line per
entry", plain flat text.

It's nice, but if one is wanting to push and index those logs in an ELK,
the current, default format isn't really good.

After some discussions about oslo.log, it appears it provides a nice
JSONFormatter handler¹ one might want to use for each (python) service
using oslo central library.

A JSON format is really cool, as it's easy to parse for machines, and it
can be on a multi-line without any bit issue - this is especially
important for stack traces, as their output is multi-line without real
way to have a common delimiter so that we can re-format it and feed it
to any log parser (logstash, fluentd, …).

After some more talks, olso.log will not provide a unified interface in
order to output all received logs as JSON - this makes sens, as that
would mean "rewrite almost all the python logging management
interface"², and that's pretty useless, since (all?) services have their
own "logging.conf" file.

That said… to the main purpose of that mail:

  • Default format for logs
    A first question would be "are we all OK with the default output format"
  • I'm pretty sure "humans" are happy with that, as it's really
    convenient to read and grep. But on a "standard" Openstack deploy, I'm
    pretty sure one does not have only one controller, one ceph node and one
    compute. Hence comes the log centralization, and with that, the log
    indexation and treatments.

For that, one might argue "I'm using plain files on my logger, and
grep-ing -r in them". That's a way to do things, and for that, plain,
flat logs are great.

But… I'm pretty sure I'm not the only one wanting to use some kind of
ELK cluster for that kind of purpose. So the right question is: what
about switching the default log format to JSON? On my part, I don't see
"cons", only "pros", but my judgment is of course biased, as I'm "alone
in my corner". But what about you, Community?

  • Provide a way to configure the output format/handler
    While poking around in the puppet modules code, I didn't find any way to
    set the output handler for the logs. For example, in puppet-nova³ we can
    set a lot of things, but not the useful handler for the output.

It would be really cool to get, for each puppet module, the capability
to set the handler so that one can just push some stuff in hiera, and
Voilà, we have JSON logs.

Doing so would allow people to chose between the default  (current)
output, and something more "computable".

Of course, either proposal will require a nice code change in all puppet
modules (add a new parameter for the foo::logging class, and use that
new param in the configuration file, and so on), but at least people
will be able to actually chose.

So, before opening an issue on each launchpad project (that would be…
long), I'd rather open the discussion in here and, eventually, come to
some nice, acceptable and accepted solution that would make the
Openstack Community happy :).

Any thoughts?

Thank you for your attention, feedback and wonderful support for that
monster project :).

Cheers,

C.

¹
https://github.com/openstack/oslo.log/blob/master/oslo_log/formatters.py#L166-L235
²
http://eavesdrop.openstack.org/irclogs/%23openstack-oslo/%23openstack-oslo.2017-11-01.log.html#t2017-11-01T13:23:14
³ https://github.com/openstack/puppet-nova/blob/master/manifests/logging.pp

--
Cédric Jeanneret
Senior Linux System Administrator
Infrastructure Solutions

Camptocamp SA
PSE-A / EPFL, 1015 Lausanne

Phone: +41 21 619 10 32
Office: +41 21 619 10 02
Email: cedric.jeanneret@camptocamp.com


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

asked Nov 6, 2017 in openstack-dev by Cédric_Jeanneret (240 points)  

15 Responses

0 votes

Hi. Few comments inline.

Dear Stackers,
While working on my locally deployed Openstack (Pike using TripleO), I
was a bit struggling with the logging part. Currently, all logs are
pushed to per-service files, in the standard format "one line per
entry", plain flat text.

It's nice, but if one is wanting to push and index those logs in an ELK,
the current, default format isn't really good.

After some discussions about oslo.log, it appears it provides a nice
JSONFormatter handler¹ one might want to use for each (python) service
using oslo central library.

A JSON format is really cool, as it's easy to parse for machines, and it
can be on a multi-line without any bit issue - this is especially
important for stack traces, as their output is multi-line without real
way to have a common delimiter so that we can re-format it and feed it
to any log parser (logstash, fluentd, …).

After some more talks, olso.log will not provide a unified interface in
order to output all received logs as JSON - this makes sens, as that
would mean "rewrite almost all the python logging management
interface"², and that's pretty useless, since (all?) services have their
own "logging.conf" file.

That said… to the main purpose of that mail:

  • Default format for logs
    A first question would be "are we all OK with the default output format"
  • I'm pretty sure "humans" are happy with that, as it's really
    convenient to read and grep. But on a "standard" Openstack deploy, I'm
    pretty sure one does not have only one controller, one ceph node and one
    compute. Hence comes the log centralization, and with that, the log
    indexation and treatments.

For that, one might argue "I'm using plain files on my logger, and
grep-ing -r in them". That's a way to do things, and for that, plain,
flat logs are great.

But… I'm pretty sure I'm not the only one wanting to use some kind of
ELK cluster for that kind of purpose. So the right question is: what
about switching the default log format to JSON? On my part, I don't see
"cons", only "pros", but my judgment is of course biased, as I'm "alone
in my corner". But what about you, Community?

  • Provide a way to configure the output format/handler
    While poking around in the puppet modules code, I didn't find any way to
    set the output handler for the logs. For example, in puppet-nova³ we can
    set a lot of things, but not the useful handler for the output.

It would be really cool to get, for each puppet module, the capability
to set the handler so that one can just push some stuff in hiera, and
Voilà, we have JSON logs.

Doing so would allow people to chose between the default (current)
output, and something more "computable".

This is being implemented (for Q) in the scope of this blueprint [0].
By default, containers will keep the current logging behaviour, which is
writing logs into bind-mounted host-path directories, and for those
services that can syslog only, sending the logs to the host journald via
syslog (bind-mounted /dev/log). And if the stdout/stderr feature
enabled, side-car containers will make sure everything is captured in
journald and tagged properly, with multiline entries as well. And
journald can dump messages as JSON as well. I hope that works! Please
send you comments to the aforementioned bp's spec.

[0] https://blueprints.launchpad.net/tripleo/+spec/logging-stdout-rsyslog

Of course, either proposal will require a nice code change in all puppet
modules (add a new parameter for the foo::logging class, and use that
new param in the configuration file, and so on), but at least people
will be able to actually chose.

So, before opening an issue on each launchpad project (that would be…
long), I'd rather open the discussion in here and, eventually, come to
some nice, acceptable and accepted solution that would make the
Openstack Community happy :).

Any thoughts?

Thank you for your attention, feedback and wonderful support for that
monster project :).

Cheers,

C.

¹
https://github.com/openstack/oslo.log/blob/master/oslo_log/formatters.py#L166-L235
²
http://eavesdrop.openstack.org/irclogs/%23openstack-oslo/%23openstack-oslo.2017-11-01.log.html#t2017-11-01T13:23:14
³ https://github.com/openstack/puppet-nova/blob/master/manifests/logging.pp

--
Cédric Jeanneret
Senior Linux System Administrator
Infrastructure Solutions

Camptocamp SA
PSE-A / EPFL, 1015 Lausanne

Phone: +41 21 619 10 32
Office: +41 21 619 10 02
Email: cedric.jeanneret@camptocamp.com

--
Best regards,
Bogdan Dobrelya,
Irc #bogdando


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 2, 2017 by bdobreli_at_redhat.c (2,260 points)   2 3
0 votes

On Thu, Nov 2, 2017 at 11:15 AM, Bogdan Dobrelya bdobreli@redhat.com
wrote:

Hi. Few comments inline.

Dear Stackers,

While working on my locally deployed Openstack (Pike using TripleO), I
was a bit struggling with the logging part. Currently, all logs are
pushed to per-service files, in the standard format "one line per
entry", plain flat text.

It's nice, but if one is wanting to push and index those logs in an ELK,
the current, default format isn't really good.

After some discussions about oslo.log, it appears it provides a nice
JSONFormatter handler¹ one might want to use for each (python) service
using oslo central library.

A JSON format is really cool, as it's easy to parse for machines, and it
can be on a multi-line without any bit issue - this is especially
important for stack traces, as their output is multi-line without real
way to have a common delimiter so that we can re-format it and feed it
to any log parser (logstash, fluentd, …).

After some more talks, olso.log will not provide a unified interface in
order to output all received logs as JSON - this makes sens, as that
would mean "rewrite almost all the python logging management
interface"², and that's pretty useless, since (all?) services have their
own "logging.conf" file.

That said… to the main purpose of that mail:

  • Default format for logs
    A first question would be "are we all OK with the default output format"
  • I'm pretty sure "humans" are happy with that, as it's really
    convenient to read and grep. But on a "standard" Openstack deploy, I'm
    pretty sure one does not have only one controller, one ceph node and one
    compute. Hence comes the log centralization, and with that, the log
    indexation and treatments.

For that, one might argue "I'm using plain files on my logger, and
grep-ing -r in them". That's a way to do things, and for that, plain,
flat logs are great.

But… I'm pretty sure I'm not the only one wanting to use some kind of
ELK cluster for that kind of purpose. So the right question is: what
about switching the default log format to JSON? On my part, I don't see
"cons", only "pros", but my judgment is of course biased, as I'm "alone
in my corner". But what about you, Community?

  • Provide a way to configure the output format/handler
    While poking around in the puppet modules code, I didn't find any way to
    set the output handler for the logs. For example, in puppet-nova³ we can
    set a lot of things, but not the useful handler for the output.

Right, I was also looking into this and am yet to find how it's done. Any
idea how?

It would be really cool to get, for each puppet module, the capability
to set the handler so that one can just push some stuff in hiera, and
Voilà, we have JSON logs.

Doing so would allow people to chose between the default (current)
output, and something more "computable".

This is being implemented (for Q) in the scope of this blueprint [0].
By default, containers will keep the current logging behaviour, which is
writing logs into bind-mounted host-path directories, and for those
services that can syslog only, sending the logs to the host journald via
syslog (bind-mounted /dev/log). And if the stdout/stderr feature enabled,
side-car containers will make sure everything is captured in journald and
tagged properly, with multiline entries as well. And journald can dump
messages as JSON as well. I hope that works! Please send you comments to
the aforementioned bp's spec.

[0] https://blueprints.launchpad.net/tripleo/+spec/logging-stdout-rsyslog

Correct, as Bogdan mentioned, that will hopefully come as part of the work
of that blueprint. The plan so far is to keep the logs as they currently
are and then put them in JSON format in an rsyslog forwarder (the
implementation is still to come).

However, one thing that would be great would be to have oslo.log already
provide JSON output, that way we would need way less processing. And most
importantly, it would make StackTrace processing easier. Right now it's a
pain to reconstruct python exceptions as they end up being forwarded as
several log lines, and having it as one line, or as JSON output already
would be great. Any idea if this is possible already via oslo.log?

Of course, either proposal will require a nice code change in all puppet
modules (add a new parameter for the foo::logging class, and use that
new param in the configuration file, and so on), but at least people
will be able to actually chose.

So, before opening an issue on each launchpad project (that would be…
long), I'd rather open the discussion in here and, eventually, come to
some nice, acceptable and accepted solution that would make the
Openstack Community happy :).

Any thoughts?

Thank you for your attention, feedback and wonderful support for that
monster project :).

Cheers,

C.

¹
https://github.com/openstack/oslo.log/blob/master/oslo_log/f
ormatters.py#L166-L235
²
http://eavesdrop.openstack.org/irclogs/%23openstack-oslo/%
23openstack-oslo.2017-11-01.log.html#t2017-11-01T13:23:14
³ https://github.com/openstack/puppet-nova/blob/master/manifes
ts/logging.pp

--
Cédric Jeanneret
Senior Linux System Administrator
Infrastructure Solutions

Camptocamp SA
PSE-A / EPFL, 1015 Lausanne

Phone: +41 21 619 10 32
Office: +41 21 619 10 02
Email: cedric.jeanneret@camptocamp.com

--
Best regards,
Bogdan Dobrelya,
Irc #bogdando


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Juan Antonio Osorio R.
e-mail: jaosorior@gmail.com


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 2, 2017 by Juan_Antonio_Osorio (2,900 points)   1 2
0 votes

Adding tags for the relevant projects. I think this is mostly
directed at Puppet/TripleO, but Oslo is obviously relevant as well.

On 11/01/2017 08:54 AM, Cédric Jeanneret wrote:
Dear Stackers,

While working on my locally deployed Openstack (Pike using TripleO), I
was a bit struggling with the logging part. Currently, all logs are
pushed to per-service files, in the standard format "one line per
entry", plain flat text.

It's nice, but if one is wanting to push and index those logs in an ELK,
the current, default format isn't really good.

After some discussions about oslo.log, it appears it provides a nice
JSONFormatter handler¹ one might want to use for each (python) service
using oslo central library.

A JSON format is really cool, as it's easy to parse for machines, and it
can be on a multi-line without any bit issue - this is especially
important for stack traces, as their output is multi-line without real
way to have a common delimiter so that we can re-format it and feed it
to any log parser (logstash, fluentd, …).

After some more talks, olso.log will not provide a unified interface in
order to output all received logs as JSON - this makes sens, as that
would mean "rewrite almost all the python logging management
interface"², and that's pretty useless, since (all?) services have their
own "logging.conf" file.

That said… to the main purpose of that mail:

  • Default format for logs
    A first question would be "are we all OK with the default output format"
  • I'm pretty sure "humans" are happy with that, as it's really
    convenient to read and grep. But on a "standard" Openstack deploy, I'm
    pretty sure one does not have only one controller, one ceph node and one
    compute. Hence comes the log centralization, and with that, the log
    indexation and treatments.

For that, one might argue "I'm using plain files on my logger, and
grep-ing -r in them". That's a way to do things, and for that, plain,
flat logs are great.

But… I'm pretty sure I'm not the only one wanting to use some kind of
ELK cluster for that kind of purpose. So the right question is: what
about switching the default log format to JSON? On my part, I don't see
"cons", only "pros", but my judgment is of course biased, as I'm "alone
in my corner". But what about you, Community?

The major con I see is that we don't require an ELK stack and reading
JSON logs if you don't have one of those is probably worse, although I
will admit I haven't spent much time reading our JSON-formatted logs.
My experience with things that log in JSON has not generally been
positive though. It's just not as human-readable.

The other problem with changing log format defaults is that many people
have already set up filters and processing based on the existing log
format. There are significant user impacts to changing that default.

  • Provide a way to configure the output format/handler
    While poking around in the puppet modules code, I didn't find any way to
    set the output handler for the logs. For example, in puppet-nova³ we can
    set a lot of things, but not the useful handler for the output.

It would be really cool to get, for each puppet module, the capability
to set the handler so that one can just push some stuff in hiera, and
Voilà, we have JSON logs.

Doing so would allow people to chose between the default  (current)
output, and something more "computable".

I think we should do this regardless. There are valid arguments for
people to want both log formats IMHO and we should allow people to use
what they want.

Of course, either proposal will require a nice code change in all puppet
modules (add a new parameter for the foo::logging class, and use that
new param in the configuration file, and so on), but at least people
will be able to actually chose.

So, before opening an issue on each launchpad project (that would be…
long), I'd rather open the discussion in here and, eventually, come to
some nice, acceptable and accepted solution that would make the
Openstack Community happy :).

Any thoughts?

Thank you for your attention, feedback and wonderful support for that
monster project :).

Cheers,

C.

¹
https://github.com/openstack/oslo.log/blob/master/oslo_log/formatters.py#L166-L235
²
http://eavesdrop.openstack.org/irclogs/%23openstack-oslo/%23openstack-oslo.2017-11-01.log.html#t2017-11-01T13:23:14
³ https://github.com/openstack/puppet-nova/blob/master/manifests/logging.pp


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 2, 2017 by Ben_Nemec (19,660 points)   2 3 3
0 votes

On 11/02/2017 05:18 PM, Ben Nemec wrote:
Adding tags for the relevant projects.  I think this is mostly
directed at Puppet/TripleO, but Oslo is obviously relevant as well.

Thank you! First mail in there, wasn't really sure how to do that.

On 11/01/2017 08:54 AM, Cédric Jeanneret wrote:

Dear Stackers,

While working on my locally deployed Openstack (Pike using TripleO), I
was a bit struggling with the logging part. Currently, all logs are
pushed to per-service files, in the standard format "one line per
entry", plain flat text.

It's nice, but if one is wanting to push and index those logs in an ELK,
the current, default format isn't really good.

After some discussions about oslo.log, it appears it provides a nice
JSONFormatter handler¹ one might want to use for each (python) service
using oslo central library.

A JSON format is really cool, as it's easy to parse for machines, and it
can be on a multi-line without any bit issue - this is especially
important for stack traces, as their output is multi-line without real
way to have a common delimiter so that we can re-format it and feed it
to any log parser (logstash, fluentd, …).

After some more talks, olso.log will not provide a unified interface in
order to output all received logs as JSON - this makes sens, as that
would mean "rewrite almost all the python logging management
interface"², and that's pretty useless, since (all?) services have their
own "logging.conf" file.

That said… to the main purpose of that mail:

  • Default format for logs
    A first question would be "are we all OK with the default output format"
  • I'm pretty sure "humans" are happy with that, as it's really
    convenient to read and grep. But on a "standard" Openstack deploy, I'm
    pretty sure one does not have only one controller, one ceph node and one
    compute. Hence comes the log centralization, and with that, the log
    indexation and treatments.

For that, one might argue "I'm using plain files on my logger, and
grep-ing -r in them". That's a way to do things, and for that, plain,
flat logs are great.

But… I'm pretty sure I'm not the only one wanting to use some kind of
ELK cluster for that kind of purpose. So the right question is: what
about switching the default log format to JSON? On my part, I don't see
"cons", only "pros", but my judgment is of course biased, as I'm "alone
in my corner". But what about you, Community?

The major con I see is that we don't require an ELK stack and reading
JSON logs if you don't have one of those is probably worse, although I
will admit I haven't spent much time reading our JSON-formatted logs. My
experience with things that log in JSON has not generally been positive
though.  It's just not as human-readable.

We're on the same line on that - hence the following proposal. I was
pretty sure switching the default format was a bad thing, but I had to
submit it in order to be complete ;).
Let's focus on the other one, as it's more friendly for everyone.

The other problem with changing log format defaults is that many people
have already set up filters and processing based on the existing log
format.  There are significant user impacts to changing that default.

  • Provide a way to configure the output format/handler
    While poking around in the puppet modules code, I didn't find any way to
    set the output handler for the logs. For example, in puppet-nova³ we can
    set a lot of things, but not the useful handler for the output.

It would be really cool to get, for each puppet module, the capability
to set the handler so that one can just push some stuff in hiera, and
Voilà, we have JSON logs.

Doing so would allow people to chose between the default  (current)
output, and something more "computable".

I think we should do this regardless.  There are valid arguments for
people to want both log formats IMHO and we should allow people to use
what they want.

If I understand the things correctly, that would require a "small"
change in every puppet modules so that it configures the service with
the proper log output. Any way to automate something on that? It might
be worth to do some PoC on a specific module maybe?

Of course, either proposal will require a nice code change in all puppet
modules (add a new parameter for the foo::logging class, and use that
new param in the configuration file, and so on), but at least people
will be able to actually chose.

So, before opening an issue on each launchpad project (that would be…
long), I'd rather open the discussion in here and, eventually, come to
some nice, acceptable and accepted solution that would make the
Openstack Community happy :).

Any thoughts?

Thank you for your attention, feedback and wonderful support for that
monster project :).

Cheers,

C.

¹
https://github.com/openstack/oslo.log/blob/master/oslo_log/formatters.py#L166-L235

²
http://eavesdrop.openstack.org/irclogs/%23openstack-oslo/%23openstack-oslo.2017-11-01.log.html#t2017-11-01T13:23:14

³
https://github.com/openstack/puppet-nova/blob/master/manifests/logging.pp


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

responded Nov 3, 2017 by Cédric_Jeanneret (240 points)  
0 votes

On Fri, Nov 3, 2017 at 12:21 AM, Cédric Jeanneret
cedric.jeanneret@camptocamp.com wrote:
On 11/02/2017 05:18 PM, Ben Nemec wrote:

Adding tags for the relevant projects. I think this is mostly
directed at Puppet/TripleO, but Oslo is obviously relevant as well.

Thank you! First mail in there, wasn't really sure how to do that.

On 11/01/2017 08:54 AM, Cédric Jeanneret wrote:

Dear Stackers,

While working on my locally deployed Openstack (Pike using TripleO), I
was a bit struggling with the logging part. Currently, all logs are
pushed to per-service files, in the standard format "one line per
entry", plain flat text.

It's nice, but if one is wanting to push and index those logs in an ELK,
the current, default format isn't really good.

After some discussions about oslo.log, it appears it provides a nice
JSONFormatter handler¹ one might want to use for each (python) service
using oslo central library.

A JSON format is really cool, as it's easy to parse for machines, and it
can be on a multi-line without any bit issue - this is especially
important for stack traces, as their output is multi-line without real
way to have a common delimiter so that we can re-format it and feed it
to any log parser (logstash, fluentd, …).

After some more talks, olso.log will not provide a unified interface in
order to output all received logs as JSON - this makes sens, as that
would mean "rewrite almost all the python logging management
interface"², and that's pretty useless, since (all?) services have their
own "logging.conf" file.

That said… to the main purpose of that mail:

  • Default format for logs
    A first question would be "are we all OK with the default output format"
  • I'm pretty sure "humans" are happy with that, as it's really
    convenient to read and grep. But on a "standard" Openstack deploy, I'm
    pretty sure one does not have only one controller, one ceph node and one
    compute. Hence comes the log centralization, and with that, the log
    indexation and treatments.

For that, one might argue "I'm using plain files on my logger, and
grep-ing -r in them". That's a way to do things, and for that, plain,
flat logs are great.

But… I'm pretty sure I'm not the only one wanting to use some kind of
ELK cluster for that kind of purpose. So the right question is: what
about switching the default log format to JSON? On my part, I don't see
"cons", only "pros", but my judgment is of course biased, as I'm "alone
in my corner". But what about you, Community?

The major con I see is that we don't require an ELK stack and reading
JSON logs if you don't have one of those is probably worse, although I
will admit I haven't spent much time reading our JSON-formatted logs. My
experience with things that log in JSON has not generally been positive
though. It's just not as human-readable.

We're on the same line on that - hence the following proposal. I was
pretty sure switching the default format was a bad thing, but I had to
submit it in order to be complete ;).
Let's focus on the other one, as it's more friendly for everyone.

The other problem with changing log format defaults is that many people
have already set up filters and processing based on the existing log
format. There are significant user impacts to changing that default.

  • Provide a way to configure the output format/handler
    While poking around in the puppet modules code, I didn't find any way to
    set the output handler for the logs. For example, in puppet-nova³ we can
    set a lot of things, but not the useful handler for the output.

It would be really cool to get, for each puppet module, the capability
to set the handler so that one can just push some stuff in hiera, and
Voilà, we have JSON logs.

Doing so would allow people to chose between the default (current)
output, and something more "computable".

I think we should do this regardless. There are valid arguments for
people to want both log formats IMHO and we should allow people to use
what they want.

If I understand the things correctly, that would require a "small"
change in every puppet modules so that it configures the service with
the proper log output. Any way to automate something on that? It might
be worth to do some PoC on a specific module maybe?

So for the puppet modules, the available logging configuration lives
in puppet-oslo so if there's a configuration around that that needs to
be updated[0] then that is the first change. From there you would need
to go through each module to expose this new configuration in the
logging classes. For example puppet-nova[1]. You could probably write
a script to modify each but I'm not sure they are completely
consistent between classes. I think they are but it would require some
validation.

[0] https://github.com/openstack/puppet-oslo/blob/master/manifests/log.pp
[1] https://github.com/openstack/puppet-nova/blob/master/manifests/logging.pp

Of course, either proposal will require a nice code change in all puppet
modules (add a new parameter for the foo::logging class, and use that
new param in the configuration file, and so on), but at least people
will be able to actually chose.

So, before opening an issue on each launchpad project (that would be…
long), I'd rather open the discussion in here and, eventually, come to
some nice, acceptable and accepted solution that would make the
Openstack Community happy :).

Any thoughts?

Thank you for your attention, feedback and wonderful support for that
monster project :).

Cheers,

C.

¹
https://github.com/openstack/oslo.log/blob/master/oslo_log/formatters.py#L166-L235

²
http://eavesdrop.openstack.org/irclogs/%23openstack-oslo/%23openstack-oslo.2017-11-01.log.html#t2017-11-01T13:23:14

³
https://github.com/openstack/puppet-nova/blob/master/manifests/logging.pp


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 3, 2017 by aschultz_at_redhat.c (5,800 points)   2 2 4
0 votes

Excerpts from Cédric Jeanneret's message of 2017-11-01 14:54:34 +0100:

Dear Stackers,

While working on my locally deployed Openstack (Pike using TripleO), I
was a bit struggling with the logging part. Currently, all logs are
pushed to per-service files, in the standard format "one line per
entry", plain flat text.

It's nice, but if one is wanting to push and index those logs in an ELK,
the current, default format isn't really good.

After some discussions about oslo.log, it appears it provides a nice
JSONFormatter handler¹ one might want to use for each (python) service
using oslo central library.

A JSON format is really cool, as it's easy to parse for machines, and it
can be on a multi-line without any bit issue - this is especially
important for stack traces, as their output is multi-line without real
way to have a common delimiter so that we can re-format it and feed it
to any log parser (logstash, fluentd, …).

After some more talks, olso.log will not provide a unified interface in
order to output all received logs as JSON - this makes sens, as that
would mean "rewrite almost all the python logging management
interface"², and that's pretty useless, since (all?) services have their
own "logging.conf" file.

That said… to the main purpose of that mail:

  • Default format for logs
    A first question would be "are we all OK with the default output format"
  • I'm pretty sure "humans" are happy with that, as it's really
    convenient to read and grep. But on a "standard" Openstack deploy, I'm
    pretty sure one does not have only one controller, one ceph node and one
    compute. Hence comes the log centralization, and with that, the log
    indexation and treatments.

For that, one might argue "I'm using plain files on my logger, and
grep-ing -r in them". That's a way to do things, and for that, plain,
flat logs are great.

But… I'm pretty sure I'm not the only one wanting to use some kind of
ELK cluster for that kind of purpose. So the right question is: what
about switching the default log format to JSON? On my part, I don't see
"cons", only "pros", but my judgment is of course biased, as I'm "alone
in my corner". But what about you, Community?

  • Provide a way to configure the output format/handler
    While poking around in the puppet modules code, I didn't find any way to
    set the output handler for the logs. For example, in puppet-nova³ we can
    set a lot of things, but not the useful handler for the output.

It would be really cool to get, for each puppet module, the capability
to set the handler so that one can just push some stuff in hiera, and
Voilà, we have JSON logs.

Doing so would allow people to chose between the default  (current)
output, and something more "computable".

Using the JSON formatter currently requires setting a logging
configuration file using the standard library configuration format
and fully specifying things like log levels, handlers, and output
destination. Would it make sense to add an option in oslo.log to
give deployers an easier way to enable the JSON formatter?

Doug

Of course, either proposal will require a nice code change in all puppet
modules (add a new parameter for the foo::logging class, and use that
new param in the configuration file, and so on), but at least people
will be able to actually chose.

So, before opening an issue on each launchpad project (that would be…
long), I'd rather open the discussion in here and, eventually, come to
some nice, acceptable and accepted solution that would make the
Openstack Community happy :).

Any thoughts?

Thank you for your attention, feedback and wonderful support for that
monster project :).

Cheers,

C.

¹
https://github.com/openstack/oslo.log/blob/master/oslo_log/formatters.py#L166-L235
²
http://eavesdrop.openstack.org/irclogs/%23openstack-oslo/%23openstack-oslo.2017-11-01.log.html#t2017-11-01T13:23:14
³ https://github.com/openstack/puppet-nova/blob/master/manifests/logging.pp


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 3, 2017 by Doug_Hellmann (87,520 points)   3 4 5
0 votes

On 3 Nov 2017 19:59, "Doug Hellmann" doug@doughellmann.com wrote:

Excerpts from Cédric Jeanneret's message of 2017-11-01 14:54:34 +0100:

Dear Stackers,

While working on my locally deployed Openstack (Pike using TripleO), I
was a bit struggling with the logging part. Currently, all logs are
pushed to per-service files, in the standard format "one line per
entry", plain flat text.

It's nice, but if one is wanting to push and index those logs in an ELK,
the current, default format isn't really good.

After some discussions about oslo.log, it appears it provides a nice
JSONFormatter handler¹ one might want to use for each (python) service
using oslo central library.

A JSON format is really cool, as it's easy to parse for machines, and it
can be on a multi-line without any bit issue - this is especially
important for stack traces, as their output is multi-line without real
way to have a common delimiter so that we can re-format it and feed it
to any log parser (logstash, fluentd, …).

After some more talks, olso.log will not provide a unified interface in
order to output all received logs as JSON - this makes sens, as that
would mean "rewrite almost all the python logging management
interface"², and that's pretty useless, since (all?) services have their
own "logging.conf" file.

That said… to the main purpose of that mail:

  • Default format for logs
    A first question would be "are we all OK with the default output format"
  • I'm pretty sure "humans" are happy with that, as it's really
    convenient to read and grep. But on a "standard" Openstack deploy, I'm
    pretty sure one does not have only one controller, one ceph node and one
    compute. Hence comes the log centralization, and with that, the log
    indexation and treatments.

For that, one might argue "I'm using plain files on my logger, and
grep-ing -r in them". That's a way to do things, and for that, plain,
flat logs are great.

But… I'm pretty sure I'm not the only one wanting to use some kind of
ELK cluster for that kind of purpose. So the right question is: what
about switching the default log format to JSON? On my part, I don't see
"cons", only "pros", but my judgment is of course biased, as I'm "alone
in my corner". But what about you, Community?

  • Provide a way to configure the output format/handler
    While poking around in the puppet modules code, I didn't find any way to
    set the output handler for the logs. For example, in puppet-nova³ we can
    set a lot of things, but not the useful handler for the output.

It would be really cool to get, for each puppet module, the capability
to set the handler so that one can just push some stuff in hiera, and
Voilà, we have JSON logs.

Doing so would allow people to chose between the default (current)
output, and something more "computable".

Using the JSON formatter currently requires setting a logging
configuration file using the standard library configuration format
and fully specifying things like log levels, handlers, and output
destination. Would it make sense to add an option in oslo.log to
give deployers an easier way to enable the JSON formatter?

This would actually be very useful.

Doug

Of course, either proposal will require a nice code change in all puppet
modules (add a new parameter for the foo::logging class, and use that
new param in the configuration file, and so on), but at least people
will be able to actually chose.

So, before opening an issue on each launchpad project (that would be…
long), I'd rather open the discussion in here and, eventually, come to
some nice, acceptable and accepted solution that would make the
Openstack Community happy :).

Any thoughts?

Thank you for your attention, feedback and wonderful support for that
monster project :).

Cheers,

C.

¹
https://github.com/openstack/oslo.log/blob/master/oslo_log/
formatters.py#L166-L235
²
http://eavesdrop.openstack.org/irclogs/%23openstack-oslo/
%23openstack-oslo.2017-11-01.log.html#t2017-11-01T13:23:14
³ https://github.com/openstack/puppet-nova/blob/master/
manifests/logging.pp


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 3, 2017 by Juan_Antonio_Osorio (2,900 points)   1 2
0 votes

+1


From: Juan Antonio Osorio [jaosorior@gmail.com]
Sent: Friday, November 03, 2017 3:11 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] Logging format: let's discuss a bit about default format, format configuration and so on

On 3 Nov 2017 19:59, "Doug Hellmann" doug@doughellmann.com wrote:
Excerpts from Cédric Jeanneret's message of 2017-11-01 14:54:34 +0100:

Dear Stackers,

While working on my locally deployed Openstack (Pike using TripleO), I
was a bit struggling with the logging part. Currently, all logs are
pushed to per-service files, in the standard format "one line per
entry", plain flat text.

It's nice, but if one is wanting to push and index those logs in an ELK,
the current, default format isn't really good.

After some discussions about oslo.log, it appears it provides a nice
JSONFormatter handler¹ one might want to use for each (python) service
using oslo central library.

A JSON format is really cool, as it's easy to parse for machines, and it
can be on a multi-line without any bit issue - this is especially
important for stack traces, as their output is multi-line without real
way to have a common delimiter so that we can re-format it and feed it
to any log parser (logstash, fluentd, …).

After some more talks, olso.log will not provide a unified interface in
order to output all received logs as JSON - this makes sens, as that
would mean "rewrite almost all the python logging management
interface"², and that's pretty useless, since (all?) services have their
own "logging.conf" file.

That said… to the main purpose of that mail:

  • Default format for logs
    A first question would be "are we all OK with the default output format"
  • I'm pretty sure "humans" are happy with that, as it's really
    convenient to read and grep. But on a "standard" Openstack deploy, I'm
    pretty sure one does not have only one controller, one ceph node and one
    compute. Hence comes the log centralization, and with that, the log
    indexation and treatments.

For that, one might argue "I'm using plain files on my logger, and
grep-ing -r in them". That's a way to do things, and for that, plain,
flat logs are great.

But… I'm pretty sure I'm not the only one wanting to use some kind of
ELK cluster for that kind of purpose. So the right question is: what
about switching the default log format to JSON? On my part, I don't see
"cons", only "pros", but my judgment is of course biased, as I'm "alone
in my corner". But what about you, Community?

  • Provide a way to configure the output format/handler
    While poking around in the puppet modules code, I didn't find any way to
    set the output handler for the logs. For example, in puppet-nova³ we can
    set a lot of things, but not the useful handler for the output.

It would be really cool to get, for each puppet module, the capability
to set the handler so that one can just push some stuff in hiera, and
Voilà, we have JSON logs.

Doing so would allow people to chose between the default (current)
output, and something more "computable".

Using the JSON formatter currently requires setting a logging
configuration file using the standard library configuration format
and fully specifying things like log levels, handlers, and output
destination. Would it make sense to add an option in oslo.log to
give deployers an easier way to enable the JSON formatter?
This would actually be very useful.

Doug

Of course, either proposal will require a nice code change in all puppet
modules (add a new parameter for the foo::logging class, and use that
new param in the configuration file, and so on), but at least people
will be able to actually chose.

So, before opening an issue on each launchpad project (that would be…
long), I'd rather open the discussion in here and, eventually, come to
some nice, acceptable and accepted solution that would make the
Openstack Community happy :).

Any thoughts?

Thank you for your attention, feedback and wonderful support for that
monster project :).

Cheers,

C.

¹
https://github.com/openstack/oslo.log/blob/master/oslo_log/formatters.py#L166-L235
²
http://eavesdrop.openstack.org/irclogs/%23openstack-oslo/%23openstack-oslo.2017-11-01.log.html#t2017-11-01T13:23:14
³ https://github.com/openstack/puppet-nova/blob/master/manifests/logging.pp


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 4, 2017 by Fox,_Kevin_M (29,360 points)   1 3 4
0 votes

Excerpts from Juan Antonio Osorio's message of 2017-11-04 00:11:47 +0200:

On 3 Nov 2017 19:59, "Doug Hellmann" doug@doughellmann.com wrote:

Excerpts from Cédric Jeanneret's message of 2017-11-01 14:54:34 +0100:

Dear Stackers,

While working on my locally deployed Openstack (Pike using TripleO), I
was a bit struggling with the logging part. Currently, all logs are
pushed to per-service files, in the standard format "one line per
entry", plain flat text.

It's nice, but if one is wanting to push and index those logs in an ELK,
the current, default format isn't really good.

After some discussions about oslo.log, it appears it provides a nice
JSONFormatter handler¹ one might want to use for each (python) service
using oslo central library.

A JSON format is really cool, as it's easy to parse for machines, and it
can be on a multi-line without any bit issue - this is especially
important for stack traces, as their output is multi-line without real
way to have a common delimiter so that we can re-format it and feed it
to any log parser (logstash, fluentd, …).

After some more talks, olso.log will not provide a unified interface in
order to output all received logs as JSON - this makes sens, as that
would mean "rewrite almost all the python logging management
interface"², and that's pretty useless, since (all?) services have their
own "logging.conf" file.

That said… to the main purpose of that mail:

  • Default format for logs
    A first question would be "are we all OK with the default output format"
  • I'm pretty sure "humans" are happy with that, as it's really
    convenient to read and grep. But on a "standard" Openstack deploy, I'm
    pretty sure one does not have only one controller, one ceph node and one
    compute. Hence comes the log centralization, and with that, the log
    indexation and treatments.

For that, one might argue "I'm using plain files on my logger, and
grep-ing -r in them". That's a way to do things, and for that, plain,
flat logs are great.

But… I'm pretty sure I'm not the only one wanting to use some kind of
ELK cluster for that kind of purpose. So the right question is: what
about switching the default log format to JSON? On my part, I don't see
"cons", only "pros", but my judgment is of course biased, as I'm "alone
in my corner". But what about you, Community?

  • Provide a way to configure the output format/handler
    While poking around in the puppet modules code, I didn't find any way to
    set the output handler for the logs. For example, in puppet-nova³ we can
    set a lot of things, but not the useful handler for the output.

It would be really cool to get, for each puppet module, the capability
to set the handler so that one can just push some stuff in hiera, and
Voilà, we have JSON logs.

Doing so would allow people to chose between the default (current)
output, and something more "computable".

Using the JSON formatter currently requires setting a logging
configuration file using the standard library configuration format
and fully specifying things like log levels, handlers, and output
destination. Would it make sense to add an option in oslo.log to
give deployers an easier way to enable the JSON formatter?

This would actually be very useful.

Great! Let me know if you would like to help figuring out where to do
that in the library code.

Doug


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Nov 4, 2017 by Doug_Hellmann (87,520 points)   3 4 5
0 votes

On 11/04/2017 07:08 AM, Doug Hellmann wrote:
Excerpts from Juan Antonio Osorio's message of 2017-11-04 00:11:47 +0200:

On 3 Nov 2017 19:59, "Doug Hellmann" doug@doughellmann.com wrote:

Excerpts from Cédric Jeanneret's message of 2017-11-01 14:54:34 +0100:

Dear Stackers,

While working on my locally deployed Openstack (Pike using TripleO), I
was a bit struggling with the logging part. Currently, all logs are
pushed to per-service files, in the standard format "one line per
entry", plain flat text.

It's nice, but if one is wanting to push and index those logs in an ELK,
the current, default format isn't really good.

After some discussions about oslo.log, it appears it provides a nice
JSONFormatter handler¹ one might want to use for each (python) service
using oslo central library.

A JSON format is really cool, as it's easy to parse for machines, and it
can be on a multi-line without any bit issue - this is especially
important for stack traces, as their output is multi-line without real
way to have a common delimiter so that we can re-format it and feed it
to any log parser (logstash, fluentd, …).

After some more talks, olso.log will not provide a unified interface in
order to output all received logs as JSON - this makes sens, as that
would mean "rewrite almost all the python logging management
interface"², and that's pretty useless, since (all?) services have their
own "logging.conf" file.

That said… to the main purpose of that mail:

  • Default format for logs
    A first question would be "are we all OK with the default output format"
  • I'm pretty sure "humans" are happy with that, as it's really
    convenient to read and grep. But on a "standard" Openstack deploy, I'm
    pretty sure one does not have only one controller, one ceph node and one
    compute. Hence comes the log centralization, and with that, the log
    indexation and treatments.

For that, one might argue "I'm using plain files on my logger, and
grep-ing -r in them". That's a way to do things, and for that, plain,
flat logs are great.

But… I'm pretty sure I'm not the only one wanting to use some kind of
ELK cluster for that kind of purpose. So the right question is: what
about switching the default log format to JSON? On my part, I don't see
"cons", only "pros", but my judgment is of course biased, as I'm "alone
in my corner". But what about you, Community?

  • Provide a way to configure the output format/handler
    While poking around in the puppet modules code, I didn't find any way to
    set the output handler for the logs. For example, in puppet-nova³ we can
    set a lot of things, but not the useful handler for the output.

It would be really cool to get, for each puppet module, the capability
to set the handler so that one can just push some stuff in hiera, and
Voilà, we have JSON logs.

Doing so would allow people to chose between the default (current)
output, and something more "computable".

Using the JSON formatter currently requires setting a logging
configuration file using the standard library configuration format
and fully specifying things like log levels, handlers, and output
destination. Would it make sense to add an option in oslo.log to
give deployers an easier way to enable the JSON formatter?

This would actually be very useful.

Great! Let me know if you would like to help figuring out where to do
that in the library code.

There's already some (not really documented) feature allowing oslo.log
to load a configuration file. In fact, there's already one existing in
the keystone tree (/etc/keystone/logging.conf - altougn not loaded) - we
might base something "casual" and "generic" on that one.

I think all wsgi/python services should configure the logging through
that file so that it's clearer and cleaner. But that part is maybe a bit
too big and "aggressive" :). And the logging configuration isn't that
friendly, to be honest, at least I have some issues with its doc ;).

But I think we might come to something nice and cool. It would allow
anyone to push their own log "formatter", in the end.
So you (oslo.log) wouldn't need to work with format output requests
anymore, just provide two basics (plain and json), and let users play
with the logging configuration (and even output things in XML if they
want) ;).

Regarding oslo.log: apparently, adding the following entry in the
service configuration file should do it:
log-config-append¹

Can anyone confirm that?

¹ https://github.com/openstack/oslo.log/blob/master/oslo_log/_options.py#L44

Doug


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

responded Nov 6, 2017 by Cédric_Jeanneret (240 points)  
...