[Pacemaker] [PATCH 0/2] rsyslog/logrotate configuration snippets

[Florian Haas]

On Sun, Jan 15, 2012 at 9:27 PM, Andrew Beekhof <andrew at beekhof.net> wrote:
> On Thu, Jan 12, 2012 at 11:01 PM, Florian Haas <florian at hastexo.com> wrote:
>> On Thu, Jan 5, 2012 at 10:15 PM, Florian Haas <florian at hastexo.com> wrote:
>>> Florian Haas (2):
>>>      extra: add rsyslog configuration snippet
>>>      extra: add logrotate configuration snippet
>>>
>>>  configure.ac                      |    4 +++
>>>  extra/Makefile.am                 |    2 +-
>>>  extra/logrotate/Makefile.am       |    5 ++++
>>>  extra/logrotate/pacemaker.conf.in |    7 ++++++
>>>  extra/rsyslog/Makefile.am         |    5 ++++
>>>  extra/rsyslog/pacemaker.conf.in   |   39 +++++++++++++++++++++++++++++++++++++
>>>  6 files changed, 61 insertions(+), 1 deletions(-)
>>>  create mode 100644 extra/logrotate/Makefile.am
>>>  create mode 100644 extra/logrotate/pacemaker.conf.in
>>>  create mode 100644 extra/rsyslog/Makefile.am
>>>  create mode 100644 extra/rsyslog/pacemaker.conf.in
>>
>> Any takers on these?
>
> Sorry, I was off working on the new fencing logic and then corosync
> 2.0 support (when cman and all the plugins, including ours, go away).
>
> So a couple of comments...
>
> I fully agree that the state of our logging needs work and I can
> understand people wanting to keep the vast majority of our logs out of
> syslog.
> I'm less thrilled about one-file-per-subsystem, the cluster will often
> do a lot within a single second and splitting everything up really
> hurts the ability to correlate messages.
> I'd also suggest that /some/ information not coming directly from the
> RAs is still appropriate for syslog (such as "I'm going to move A from
> B to C" or "I'm about to turn of node D"), so the nuclear option isn't
> really thrilling me.

So everything that is logged by the RAs with ocf_log, as I wrote in
the original post, _is_ still going to whatever the default syslog
destination may be. The rsyslog config doesn't change that at all.
(Stuff that the RAs simply barf out to stdout/err would go to the lrmd
log.) I maintain that this is the stuff that is also most useful to
people. And with just that information in the syslog, you usually get
a pretty clear idea of what the heck the cluster is doing on a node,
and in what order, in about 20 lines of logs close together -- rather
than intermingled with potentially hundreds of lines of other
cluster-related log output.

And disabling the "nuclear option" is a simple means of adding a "#"
before "& ~" in the config file. You can ship it that way by default
if you think that's more appropriate. That way, people would get the
split-out logs _plus_ everything in one file, which IMHO is sometimes
very useful for pengine or lrmd troubleshooting/debugging. I,
personally, just don't want Pacemaker to flood my /var/log/messages,
so I'd definitely leave the "& ~" in there, but that may be personal
preference. I wonder what others think.

> In addition to the above distractions, I've been coming up to speed on
> libqb's logging which is opening up a lot of new doors and should
> hopefully help solve the underlying log issues.
> For starters it lets syslog/stderr/logfile all log at different levels
> of verbosity (and formats), it also supports blackboxes of which a
> dump can be triggered in response to an error condition or manually by
> the admin.
>
> The plan is something along the lines of: syslog gets NOTICE and
> above, anything else (depending on debug level and trace options) goes
> to /var/log/(cluster/?)pacemaker or whatever was configured in
> corosync.
> However, before I can enact that there will need to be an audit of the
> messages currently going to INFO (674 entries) and NOTICE(160 entries)
> with some getting bumped up, others down (possibly even to debug).
> I'd certainly be interested in feedback as to which logs should and
> should not make it.

Yes, even so, I (again, this is personal preference) would definitely
not want pengine logging (which even if half its INFO messages get
demoted to DEBUG, would still be pretty verbose) in my default
messages file.

> If you want to get analytical about it, there is also an awk script
> that I use when looking at what we log.
> I'd be interested in some numbers from the field.

Thanks; I can look at that after LCA.

Cheers,
Florian

-- 
Need help with High Availability?
http://www.hastexo.com/now