[Pacemaker] [PATCH 0/2] rsyslog/logrotate configuration snippets

[Dejan Muhamedagic]

Hi,

On Mon, Jan 16, 2012 at 04:15:20PM +1100, Andrew Beekhof wrote:
> On Mon, Jan 16, 2012 at 2:42 PM, Florian Haas <florian at hastexo.com> wrote:
> > On Mon, Jan 16, 2012 at 10:59 AM, Andrew Beekhof <andrew at beekhof.net> wrote:
> >
> >> By "Nuclear", I meant nothing at all from Pacemaker.
> >
> > Which is not what it does.
> 
> The daemons.  The RAs are not "from Pacemaker".
> This is why I wrote in my first reply:
> 
> "/some/ information not coming directly from the
> RAs is still appropriate for syslog (such as "I'm going to move A from
> B to C" or "I'm about to turn of node D")"
> 
> Where in https://github.com/fghaas/pacemaker/blob/9e9bafd44971a8f4c3cd1de62fb2278fab28489e/extra/rsyslog/pacemaker.conf.in
> does it allow any log from any daemon through?

> >> If thats what you want, there's a far easier way to achieve this and
> >> keep usable logs around for debugging, set facility to none and add a
> >> logfile.
> >
> > No, I don't want that.
> 
> So one file per pacemaker daemon is central to your proposal?

I think that we need all logs in a single file. Splitting logs
into per-daemon files doesn't seem to be of any benefit to me.
And once logs are split there's no magic putting them back
together in right order.

Furthermore, such a scheme would definitely break parts of the
crm shell and hawk history explorer. Most probably also hb_report
and crm_report.

Probably (and unfortunately) the only way is reviewing log
messages and adjusting severity. Now, I somehow doubt that we'll
ever do that, but we could perhaps solicit from users that they
send a "footprints-of-messages-that-annoy-me" kind of list, for
some of which we could then reduce severity to debug. Maybe even
invest some time into creating a web form for that.

One other option is to have one extra log file which would
contain just RA and lrmd messages. But IMO there's no replacement
for the aggregate log file.

In perfect world, we would indeed need only output of resource
and stonith agents, i.e. the cluster machinery itself should just
do what we want it to do. And eventually perhaps report that we
ran out of disk, stuff like that.

Cheers,

Dejan

> >>> (Stuff that the RAs simply barf out to stdout/err would go to the lrmd
> >>> log.) I maintain that this is the stuff that is also most useful to
> >>> people. And with just that information in the syslog, you usually get
> >>> a pretty clear idea of what the heck the cluster is doing on a node,
> >>> and in what order, in about 20 lines of logs close together -- rather
> >>> than intermingled with potentially hundreds of lines of other
> >>> cluster-related log output.
> >>
> >> Did I not just finish agreeing that "hundreds of lines of other
> >> cluster-related log[s]" was a problem?
> >
> > What in my statement above indicates that I assumed otherwise?
> 
> The part just above my reply where you're implying that the only
> alternative to your idea of removing all daemon logging from syslog
> was: resource logs "intermingled hundreds of lines of other
> cluster-related log output".
> 
> >
> >> I just don't think your knee-jerk "everything must go" approach is the answer.
> >
> > That is not my approach.
> >
> >>> And disabling the "nuclear option" is a simple means of adding a "#"
> >>> before "& ~" in the config file. You can ship it that way by default
> >>> if you think that's more appropriate. That way, people would get the
> >>> split-out logs _plus_ everything in one file, which IMHO is sometimes
> >>> very useful for pengine or lrmd troubleshooting/debugging. I,
> >>> personally, just don't want Pacemaker to flood my /var/log/messages,
> >>
> >> Did you see me arguing against that?
> >
> > No. What makes you think I did?
> 
> Because you keep trying to tell me the current approach is wrong.
> I know that, I said that, I just don't happen to think your idea is
> the solution.
> 
> >
> >>> so I'd definitely leave the "& ~" in there, but that may be personal
> >>> preference. I wonder what others think.
> >>>
> >>>> In addition to the above distractions, I've been coming up to speed on
> >>>> libqb's logging which is opening up a lot of new doors and should
> >>>> hopefully help solve the underlying log issues.
> >>>> For starters it lets syslog/stderr/logfile all log at different levels
> >>>> of verbosity (and formats), it also supports blackboxes of which a
> >>>> dump can be triggered in response to an error condition or manually by
> >>>> the admin.
> >>>>
> >>>> The plan is something along the lines of: syslog gets NOTICE and
> >>>> above, anything else (depending on debug level and trace options) goes
> >>>> to /var/log/(cluster/?)pacemaker or whatever was configured in
> >>>> corosync.
> >>>> However, before I can enact that there will need to be an audit of the
> >>>> messages currently going to INFO (674 entries) and NOTICE(160 entries)
> >>>> with some getting bumped up, others down (possibly even to debug).
> >>>> I'd certainly be interested in feedback as to which logs should and
> >>>> should not make it.
> >>>
> >>> Yes, even so, I (again, this is personal preference) would definitely
> >>> not want pengine logging (which even if half its INFO messages get
> >>> demoted to DEBUG, would still be pretty verbose) in my default
> >>> messages file.
> >>
> >> Sigh, please take time out from preaching to actually read the
> >> replies.  You might learn something.
> >
> > This is getting frustrating. Not this logging discussion, but pretty
> > much any discussion the two of us have been having lately. (And no,
> > this is not an assignment of guilt or responsibility -- it takes two
> > to tango.) Let's try and sort this out in person on Thursday.
> >
> > Florian
> >
> > _______________________________________________
> > Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> > http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> >
> > Project Home: http://www.clusterlabs.org
> > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > Bugs: http://bugs.clusterlabs.org
> 
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org