[Pacemaker] [PATCH 0/2] rsyslog/logrotate configuration snippets

[Andrew Beekhof]

On Mon, Jan 16, 2012 at 6:38 AM, Florian Haas <florian at hastexo.com> wrote:
> On Sun, Jan 15, 2012 at 9:27 PM, Andrew Beekhof <andrew at beekhof.net> wrote:
>> On Thu, Jan 12, 2012 at 11:01 PM, Florian Haas <florian at hastexo.com> wrote:
>>> On Thu, Jan 5, 2012 at 10:15 PM, Florian Haas <florian at hastexo.com> wrote:
>>>> Florian Haas (2):
>>>>      extra: add rsyslog configuration snippet
>>>>      extra: add logrotate configuration snippet
>>>>
>>>>  configure.ac                      |    4 +++
>>>>  extra/Makefile.am                 |    2 +-
>>>>  extra/logrotate/Makefile.am       |    5 ++++
>>>>  extra/logrotate/pacemaker.conf.in |    7 ++++++
>>>>  extra/rsyslog/Makefile.am         |    5 ++++
>>>>  extra/rsyslog/pacemaker.conf.in   |   39 +++++++++++++++++++++++++++++++++++++
>>>>  6 files changed, 61 insertions(+), 1 deletions(-)
>>>>  create mode 100644 extra/logrotate/Makefile.am
>>>>  create mode 100644 extra/logrotate/pacemaker.conf.in
>>>>  create mode 100644 extra/rsyslog/Makefile.am
>>>>  create mode 100644 extra/rsyslog/pacemaker.conf.in
>>>
>>> Any takers on these?
>>
>> Sorry, I was off working on the new fencing logic and then corosync
>> 2.0 support (when cman and all the plugins, including ours, go away).
>>
>> So a couple of comments...
>>
>> I fully agree that the state of our logging needs work and I can
>> understand people wanting to keep the vast majority of our logs out of
>> syslog.
>> I'm less thrilled about one-file-per-subsystem, the cluster will often
>> do a lot within a single second and splitting everything up really
>> hurts the ability to correlate messages.
>> I'd also suggest that /some/ information not coming directly from the
>> RAs is still appropriate for syslog (such as "I'm going to move A from
>> B to C" or "I'm about to turn of node D"), so the nuclear option isn't
>> really thrilling me.
>
> So everything that is logged by the RAs with ocf_log, as I wrote in
> the original post, _is_ still going to whatever the default syslog
> destination may be. The rsyslog config doesn't change that at all.

By "Nuclear", I meant nothing at all from Pacemaker.
If thats what you want, there's a far easier way to achieve this and
keep usable logs around for debugging, set facility to none and add a
logfile.

> (Stuff that the RAs simply barf out to stdout/err would go to the lrmd
> log.) I maintain that this is the stuff that is also most useful to
> people. And with just that information in the syslog, you usually get
> a pretty clear idea of what the heck the cluster is doing on a node,
> and in what order, in about 20 lines of logs close together -- rather
> than intermingled with potentially hundreds of lines of other
> cluster-related log output.

Did I not just finish agreeing that "hundreds of lines of other
cluster-related log[s]" was a problem?
I just don't think your knee-jerk "everything must go" approach is the answer.

>
> And disabling the "nuclear option" is a simple means of adding a "#"
> before "& ~" in the config file. You can ship it that way by default
> if you think that's more appropriate. That way, people would get the
> split-out logs _plus_ everything in one file, which IMHO is sometimes
> very useful for pengine or lrmd troubleshooting/debugging. I,
> personally, just don't want Pacemaker to flood my /var/log/messages,

Did you see me arguing against that?

> so I'd definitely leave the "& ~" in there, but that may be personal
> preference. I wonder what others think.
>
>> In addition to the above distractions, I've been coming up to speed on
>> libqb's logging which is opening up a lot of new doors and should
>> hopefully help solve the underlying log issues.
>> For starters it lets syslog/stderr/logfile all log at different levels
>> of verbosity (and formats), it also supports blackboxes of which a
>> dump can be triggered in response to an error condition or manually by
>> the admin.
>>
>> The plan is something along the lines of: syslog gets NOTICE and
>> above, anything else (depending on debug level and trace options) goes
>> to /var/log/(cluster/?)pacemaker or whatever was configured in
>> corosync.
>> However, before I can enact that there will need to be an audit of the
>> messages currently going to INFO (674 entries) and NOTICE(160 entries)
>> with some getting bumped up, others down (possibly even to debug).
>> I'd certainly be interested in feedback as to which logs should and
>> should not make it.
>
> Yes, even so, I (again, this is personal preference) would definitely
> not want pengine logging (which even if half its INFO messages get
> demoted to DEBUG, would still be pretty verbose) in my default
> messages file.

Sigh, please take time out from preaching to actually read the
replies.  You might learn something.
Your precious default messages file wouldn't be getting /any/ INFO
logs from pacemaker.

And I'm guessing your complaints are based on 1.0 logging too right?
Because for a long time now, the PE in 1.1 has only logged NOTICE and
above by default, which means about 1 additional line per resource +
errors/warnings (and that will soon drop to 1 per resource changing
state).

>> If you want to get analytical about it, there is also an awk script
>> that I use when looking at what we log.
>> I'd be interested in some numbers from the field.
>
> Thanks; I can look at that after LCA.
>
> Cheers,
> Florian
>
> --
> Need help with High Availability?
> http://www.hastexo.com/now
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org