[Pacemaker] pacemaker processes RSS growth

[Vladislav Bogdanov]

12.12.2012 05:35, Andrew Beekhof wrote:
> On Tue, Dec 11, 2012 at 5:49 PM, Vladislav Bogdanov
> <bubble at hoster-ok.com> wrote:
>> 11.12.2012 06:52, Vladislav Bogdanov wrote:
>>> 11.12.2012 05:12, Andrew Beekhof wrote:
>>>> On Mon, Dec 10, 2012 at 11:34 PM, Vladislav Bogdanov
>>>> <bubble at hoster-ok.com> wrote:
>>>>> 10.12.2012 09:56, Vladislav Bogdanov wrote:
>>>>>> 10.12.2012 04:29, Andrew Beekhof wrote:
>>>>>>> On Fri, Dec 7, 2012 at 5:37 PM, Vladislav Bogdanov <bubble at hoster-ok.com> wrote:
>>>>>>>> 06.12.2012 09:04, Vladislav Bogdanov wrote:
>>>>>>>>> 06.12.2012 06:05, Andrew Beekhof wrote:
>>>>>>>>>> I wonder what the growth looks like with the recent libqb fix.
>>>>>>>>>> That could be an explanation.
>>>>>>>>>
>>>>>>>>> Valid point. I will watch.
>>>>>>>>
>>>>>>>> On a almost static cluster the only change in memory state during 24
>>>>>>>> hours is +700kb of shared memory to crmd on a DC. Will look after that
>>>>>>>> one for more time.
>>>>>>
>>>>>> It still grows. ~650-700k per day. I sampled 'maps' and 'smaps' content
>>>>>> from crmd's proc and will look what differs there over the time.
>>>>>
>>>>> smaps tells me it may be in /dev/shm/qb-pengine-event-1735-1736-4-data.
>>>>> 1735 is pengine, 1736 is crmd.
>>>>>
>>>>> Diff of that part:
>>>>> @@ -56,13 +56,13 @@
>>>>>  MMUPageSize:           4 kB
>>>>>  7f427fddf000-7f42802df000 rw-s 00000000 00:0f 12332
>>>>>   /dev/shm/qb-pengine-event-1735-1736-4-data
>>>>>  Size:               5120 kB
>>>>> -Rss:                4180 kB
>>>>> -Pss:                2089 kB
>>>>> +Rss:                4320 kB
>>>>> +Pss:                2159 kB
>>>>>  Shared_Clean:          0 kB
>>>>> -Shared_Dirty:       4180 kB
>>>>> +Shared_Dirty:       4320 kB
>>>>>  Private_Clean:         0 kB
>>>>>  Private_Dirty:         0 kB
>>>>> -Referenced:         4180 kB
>>>>> +Referenced:         4320 kB
>>>>>  Anonymous:             0 kB
>>>>>  AnonHugePages:         0 kB
>>>>>  Swap:                  0 kB
>>
>> 'Rss' and 'Shared_Dirty' will soon reach 'Size' (now 4792 vs 5120), I'll
>> look what happens then. I expect growth to stop and pages to be reused.
>> If that is true, then there are no any leaks, but rather controlled fill
>> of a buffer of a predefined size.
> 
> Great. Please let me know how it turns out.

Now I see

@@ -56,13 +56,13 @@
 MMUPageSize:           4 kB
 7f427fddf000-7f42802df000 rw-s 00000000 00:0f 12332
  /dev/shm/qb-pengine-event-1735-1736-4-data
 Size:               5120 kB
-Rss:                4180 kB
-Pss:                2089 kB
+Rss:                5120 kB
+Pss:                2559 kB
 Shared_Clean:          0 kB
-Shared_Dirty:       4180 kB
+Shared_Dirty:       5120 kB
 Private_Clean:         0 kB
 Private_Dirty:         0 kB
-Referenced:         4180 kB
+Referenced:         5120 kB
 Anonymous:             0 kB
 AnonHugePages:         0 kB
 Swap:                  0 kB
@@ -70,13 +70,13 @@
 MMUPageSize:           4 kB
 7f42802df000-7f42807df000 rw-s 00000000 00:0f 12332
  /dev/shm/qb-pengine-event-1735-1736-4-data
 Size:               5120 kB
-Rss:                   0 kB
-Pss:                   0 kB
+Rss:                   4 kB
+Pss:                   1 kB
 Shared_Clean:          0 kB
-Shared_Dirty:          0 kB
+Shared_Dirty:          4 kB
 Private_Clean:         0 kB
 Private_Dirty:         0 kB
-Referenced:            0 kB
+Referenced:            4 kB
 Anonymous:             0 kB
 AnonHugePages:         0 kB
 Swap:                  0 kB

So, it stuck at 5Mb and does not grow anymore.

More, all pacemaker processes on DC for some reason now consume much
less shared memory, according to htop, than last time I looked at them.
It seems to be due to decrease of referenced pages within some anonymous
mappings. Though I do not have idea why was that happened.

Ok, the main conclusion I can make is that pacemaker does not have any
memory leaks in code paths used by a static cluster.

Will try to provide some load now.