[ClusterLabs] Antw: Re: Antw: DRBD and SSD TRIM - Slow!

[Ulrich Windl]

>>> Eric Robinson <eric.robinson at psmnv.com> schrieb am 02.08.2017 um 23:20 in
Nachricht
<DM5PR03MB2729C66CEC1E3B8B9E297185FAB00 at DM5PR03MB2729.namprd03.prod.outlook.com>

> 1) iotop did not show any significant io, just maybe 30k/second of drbd 
> traffic.
> 
> 2) okay. I've never done that before. I'll give it a shot.
> 
> 3) I'm not sure what I'm looking at there.

See /usr/src/linux/Documentation/block/stat.txt ;-)
I wrote an NRPE plugin to monitor those with performance data and verbose text output, e.g.:
CFS_VMs-xen: [delta 120s], 1.15086 IO/s read, 60.7789 IO/s write, 0 req/s read merges, 0 req/s write merges, 4.53674 sec/s read, 486.231 sec/s write, 2.36844 ms/s read wait, 2702.19 ms/s write wait, 0 req in_flight, 115.987 ms/s active, 2704.53 ms/s wait

Regards,
Ulrich

> 
> --
> Eric Robinson
> 
>> -----Original Message-----
>> From: Ulrich Windl [mailto:Ulrich.Windl at rz.uni-regensburg.de]
>> Sent: Tuesday, August 01, 2017 11:28 PM
>> To: users at clusterlabs.org 
>> Subject: [ClusterLabs] Antw: DRBD and SSD TRIM - Slow!
>> 
>> Hi!
>> 
>> I know little about trim operations, but you could try one of these:
>> 
>> 1) iotop to see whether some I/O is done during trimming (assuming
>> trimming itself is not considered to be I/O)
>> 
>> 2) Try blocktrace on the affected devices to see what's going on. It's hard 
> to
>> set up and to extract the info you are looking for, but it provides deep
>> insights
>> 
>> 3) Watch /sys/block/$BDEV/stat for performance statistics. I don't know how
>> well DRBD supports these, however (e.g. MDRAID shows no wait times and
>> no busy operations, while a multipath map has it all).
>> 
>> Regards,
>> Ulrich
>> 
>> >>> Eric Robinson <eric.robinson at psmnv.com> schrieb am 02.08.2017 um
>> >>> 07:09 in
>> Nachricht
>> <DM5PR03MB27297014DF96DC01FE849A63FAB00 at DM5PR03MB2729.nampr 
>> d03.prod.outlook.com>
>> 
>> > Does anyone know why trimming a filesystem mounted on a DRBD volume
>> > takes so long? I mean like three days to trim a 1.2TB filesystem.
>> >
>> > Here are some pertinent details:
>> >
>> > OS: SLES 12 SP2
>> > Kernel: 4.4.74-92.29
>> > Drives: 6 x Samsung SSD 840 Pro 512GB
>> > RAID: 0 (mdraid)
>> > DRBD: 9.0.8
>> > Protocol: C
>> > Network: Gigabit
>> > Utilization: 10%
>> > Latency: < 1ms
>> > Loss: 0%
>> > Iperf test: 900 mbits/sec
>> >
>> > When I write to a non-DRBD partition, I get 400MB/sec (bypassing caches).
>> > When I trim a non-DRBD partition, it completes fast.
>> > When I write to a DRBD volume, I get 80MB/sec.
>> >
>> > When I trim a DRBD volume, it takes bloody ages!
>> >
>> > --
>> > Eric Robinson
>> 
>> 
>> 
>> 
>> 
>> _______________________________________________
>> Users mailing list: Users at clusterlabs.org 
>> http://lists.clusterlabs.org/mailman/listinfo/users 
>> 
>> Project Home: http://www.clusterlabs.org Getting started:
>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf 
>> Bugs: http://bugs.clusterlabs.org 
> 
> _______________________________________________
> Users mailing list: Users at clusterlabs.org 
> http://lists.clusterlabs.org/mailman/listinfo/users 
> 
> Project Home: http://www.clusterlabs.org 
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf 
> Bugs: http://bugs.clusterlabs.org