[Pacemaker] Pacemaker still may include memory leaks

Tue May 28 00:12:09 UTC 2013

On 27/05/2013, at 5:08 PM, Vladislav Bogdanov <bubble at hoster-ok.com> wrote:

> 27.05.2013 04:20, Yuichi SEINO wrote:
>> Hi,
>> 
>> 2013/5/24 Vladislav Bogdanov <bubble at hoster-ok.com>:
>>> 24.05.2013 06:34, Andrew Beekhof wrote:
>>>> Any help figuring out where the leaks might be would be very much appreciated :)
>>> 
>>> One (and the only) suspect is unfortunately crmd itself.
>>> It has private heap grown from 2708 to 3680 kB.
>>> 
>>> All other relevant differences are in qb shm buffers, which are
>>> controlled and may grow until they reach configured size.
>>> 
>>> @Yuichi
>>> I would recommend to try running under valgrind on a testing cluster to
>>> figure out is that a memleak (lost memory) or some history data
>>> (referenced memory). Latter may be a logical memleak though. You may
>>> look in /etc/sysconfig/pacemaker for details.
>> 
>> I got valgrind for about 2 days. And, I attached valgrind in ACT node
>> and SBY node.
> 
> 
> I do not see any "direct" memory leaks (repeating 'definitely-lost'
> allocations) there.
> 
> So what we see is probably one of:
> * Cache/history/etc, which grows up to some limit (or expired at the
> some point in time).
> * Unlimited/not-expirable lists/hashes of data structures, which are
> correctly freed at exit

There is still plenty of memory chunks not free'd at exit, I'm slowly working through those.
Once we know all memory is being cleaned up, the next step is to check the size of things beforehand.

I'm hoping one or more of them show up as unnaturally large, indicating things are being added but not removed.

> (f.e like dlm_controld has(had???) for a
> debugging buffer or like glibc resolver had in EL3). This cannot be
> caught with valgrind if you use it in a standard way.
> 
> I believe we have former one. To prove that, it would be very
> interesting to run under valgrind *debugger* (--vgdb=yes|full) for some
> long enough (2-3 weeks) period of time and periodically get memory
> allocation state from there (with 'monitor leak_check full reachable
> any' gdb command). I wanted to do that a long time ago, but
> unfortunately did not have enough spare time to even try that (although
> I tried to valgrind other programs that way).
> 
> This is described in valgrind documentation:
> http://valgrind.org/docs/manual/manual-core-adv.html#manual-core-adv.gdbserver
> 
> We probably do not need to specify '--vgdb-error=0' because we do not
> need to install watchpoints at the start (and we do not need/want to
> immediately connect to crmd with gdb to tell it to continue), we just
> need to periodically get status of memory allocations
> (stop-leak_check-cont sequence). Probably that should be done in a
> 'fast' manner, so crmd does not stop for a long time, and the rest of
> pacemaker does not see it 'hanged'. Again, I did not try that, and I do
> not know if it's even possible to do that with crmd.
> 
> And, as pacemaker heavily utilizes glib, which has own memory allocator
> (slices), it is better to switch it to a 'standard' malloc/free for
> debugging with G_SLICE=always-malloc env var.
> 
> Last, I did memleak checks for a 'static' (i.e. no operations except
> monitors are performed) cluster for ~1.1.8, and did not find any. It
> would be interesting to see if that is true for an 'active' one, which
> starts/stops resources, handles failures, etc.
> 
>> 
>> Sincerely,
>> Yuichi
>> 
>>> 
>>>> 
>>>> Also, the measurements are in pages... could you run "getconf PAGESIZE" and let us know the result?
>>>> I'm guessing 4096 bytes.
>>>> 
>>>> On 23/05/2013, at 5:47 PM, Yuichi SEINO <seino.cluster2 at gmail.com> wrote:
>>>> 
>>>>> Hi,
>>>>> 
>>>>> I retry the test after we updated packages to the latest tag and OS.
>>>>> glue and booth is latest.
>>>>> 
>>>>> * Environment
>>>>> OS:RHEL 6.4
>>>>> cluster-glue:latest(commit:2755:8347e8c9b94f) +
>>>>> patch[detail:http://www.gossamer-threads.com/lists/linuxha/dev/85787]
>>>>> resource-agent:v3.9.5
>>>>> libqb:v0.14.4
>>>>> corosync:v2.3.0
>>>>> pacemaker:v1.1.10-rc2
>>>>> crmsh:v1.2.5
>>>>> booth:latest(commit:67e1208973de728958432aaba165766eac1ce3a0)
>>>>> 
>>>>> * Test procedure
>>>>> we regularly switch a ticket. The previous test also used the same way.
>>>>> And, There was no a memory leak when we tested pacemaker-1.1 before
>>>>> pacemaker use libqb.
>>>>> 
>>>>> * Result
>>>>> As a result, I think that crmd may cause the memory leak.
>>>>> 
>>>>> crmd smaps(a total of each addresses)
>>>>> In detail, we attached smaps of  start and end. And, I recorded smaps
>>>>> every 1 minutes.
>>>>> 
>>>>> Start
>>>>> RSS: 7396
>>>>> SHR(Shared_Clean+Shared_Dirty):3560
>>>>> Private(Private_Clean+Private_Dirty):3836
>>>>> 
>>>>> Interbal(about 30h later)
>>>>> RSS:18464
>>>>> SHR:14276
>>>>> Private:4188
>>>>> 
>>>>> End(about 70h later)
>>>>> RSS:19104
>>>>> SHR:14336
>>>>> Private:4768
>>>>> 
>>>>> Sincerely,
>>>>> Yuichi
>>>>> 
>>>>> 2013/5/15 Yuichi SEINO <seino.cluster2 at gmail.com>:
>>>>>> Hi,
>>>>>> 
>>>>>> I ran the test for about two days.
>>>>>> 
>>>>>> Environment
>>>>>> 
>>>>>> OS:RHEL 6.3
>>>>>> pacemaker-1.1.9-devel (commit 138556cb0b375a490a96f35e7fbeccc576a22011)
>>>>>> corosync-2.3.0
>>>>>> cluster-glue latest+patch(detail:http://www.gossamer-threads.com/lists/linuxha/dev/85787)
>>>>>> libqb- 0.14.4
>>>>>> 
>>>>>> There may be a memory leak in crmd and lrmd. I regularly got rss of ps.
>>>>>> 
>>>>>> start-up
>>>>>> crmd:5332
>>>>>> lrmd:3625
>>>>>> 
>>>>>> interval(about 30h later)
>>>>>> crmd:7716
>>>>>> lrmd:3744
>>>>>> 
>>>>>> ending(about 60h later)
>>>>>> crmd:8336
>>>>>> lrmd:3780
>>>>>> 
>>>>>> I still don't run a test that pacemaker-1.1.10-rc2 use. So, I will run its test.
>>>>>> 
>>>>>> Sincerely,
>>>>>> Yuichi
>>>>>> 
>>>>>> --
>>>>>> Yuichi SEINO
>>>>>> METROSYSTEMS CORPORATION
>>>>>> E-mail:seino.cluster2 at gmail.com
>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> Yuichi SEINO
>>>>> METROSYSTEMS CORPORATION
>>>>> E-mail:seino.cluster2 at gmail.com
>>>>> <smaps_log.tar.gz>_______________________________________________
>>>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>>> 
>>>>> Project Home: http://www.clusterlabs.org
>>>>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>> Bugs: http://bugs.clusterlabs.org
>>>> 
>>>> 
>>>> _______________________________________________
>>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>> 
>>>> Project Home: http://www.clusterlabs.org
>>>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>> Bugs: http://bugs.clusterlabs.org
>>>> 
>>> 
>>> 
>>> _______________________________________________
>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>> 
>>> Project Home: http://www.clusterlabs.org
>>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>> Bugs: http://bugs.clusterlabs.org
>> 
>> 
>> 
>> --
>> Yuichi SEINO
>> METROSYSTEMS CORPORATION
>> E-mail:seino.cluster2 at gmail.com
>> 
>> 
>> 
>> _______________________________________________
>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>> 
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>> 
> 
> 
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org