Demonstrations of compactstall, the Linux eBPF/bcc version.


compactsnoop traces the compact zone system-wide, and print various details.
Example output (manual trigger by echo 1 > /proc/sys/vm/compact_memory):

# ./compactsnoop
COMM           PID    NODE ZONE         ORDER MODE      LAT(ms)           STATUS
zsh            23685  0    ZONE_DMA     -1    SYNC        0.025         complete
zsh            23685  0    ZONE_DMA32   -1    SYNC        3.925         complete
zsh            23685  0    ZONE_NORMAL  -1    SYNC      113.975         complete
zsh            23685  1    ZONE_NORMAL  -1    SYNC        81.57         complete
zsh            23685  0    ZONE_DMA     -1    SYNC         0.02         complete
zsh            23685  0    ZONE_DMA32   -1    SYNC        4.631         complete
zsh            23685  0    ZONE_NORMAL  -1    SYNC      113.975         complete
zsh            23685  1    ZONE_NORMAL  -1    SYNC       80.647         complete
zsh            23685  0    ZONE_DMA     -1    SYNC        0.020         complete
zsh            23685  0    ZONE_DMA32   -1    SYNC        3.367         complete
zsh            23685  0    ZONE_NORMAL  -1    SYNC       115.18         complete
zsh            23685  1    ZONE_NORMAL  -1    SYNC       81.766         complete
zsh            23685  0    ZONE_DMA     -1    SYNC        0.025         complete
zsh            23685  0    ZONE_DMA32   -1    SYNC        4.346         complete
zsh            23685  0    ZONE_NORMAL  -1    SYNC      114.570         complete
zsh            23685  1    ZONE_NORMAL  -1    SYNC       80.820         complete
zsh            23685  0    ZONE_DMA     -1    SYNC        0.026         complete
zsh            23685  0    ZONE_DMA32   -1    SYNC        4.611         complete
zsh            23685  0    ZONE_NORMAL  -1    SYNC      113.993         complete
zsh            23685  1    ZONE_NORMAL  -1    SYNC       80.928         complete
zsh            23685  0    ZONE_DMA     -1    SYNC         0.02         complete
zsh            23685  0    ZONE_DMA32   -1    SYNC        3.889         complete
zsh            23685  0    ZONE_NORMAL  -1    SYNC      113.776         complete
zsh            23685  1    ZONE_NORMAL  -1    SYNC       80.727         complete
^C

While tracing, the processes alloc pages due to memory fragmentation is too
serious to meet contiguous memory requirements in the system, compact zone
events happened, which will increase the waiting delay of the processes.

compactsnoop can be useful for discovering when compact_stall(/proc/vmstat)
continues to increase, whether it is caused by some critical processes or not.

The STATUS include (CentOS 7.6's kernel)

    compact_status = {
        # COMPACT_SKIPPED: compaction didn't start as it was not possible or direct reclaim was more suitable
        0: "skipped",
        # COMPACT_CONTINUE: compaction should continue to another pageblock
        1: "continue",
        # COMPACT_PARTIAL: direct compaction partially compacted a zone and there are suitable pages
        2: "partial",
        # COMPACT_COMPLETE: The full zone was compacted
        3: "complete",
    }

or (kernel 4.7 and above)

    compact_status = {
        # COMPACT_NOT_SUITABLE_ZONE: For more detailed tracepoint output - internal to compaction
        0: "not_suitable_zone",
        # COMPACT_SKIPPED: compaction didn't start as it was not possible or direct reclaim was more suitable
        1: "skipped",
        # COMPACT_DEFERRED: compaction didn't start as it was deferred due to past failures
        2: "deferred",
        # COMPACT_NOT_SUITABLE_PAGE: For more detailed tracepoint output - internal to compaction
        3: "no_suitable_page",
        # COMPACT_CONTINUE: compaction should continue to another pageblock
        4: "continue",
        # COMPACT_COMPLETE: The full zone was compacted scanned but wasn't successful to compact suitable pages.
        5: "complete",
        # COMPACT_PARTIAL_SKIPPED: direct compaction has scanned part of the zone but wasn't successful to compact suitable pages.
        6: "partial_skipped",
        # COMPACT_CONTENDED: compaction terminated prematurely due to lock contentions
        7: "contended",
        # COMPACT_SUCCESS: direct compaction terminated after concluding that the allocation should now succeed
        8: "success",
    }

The -p option can be used to filter on a PID, which is filtered in-kernel. Here
I've used it with -T to print timestamps:

# ./compactsnoop -Tp 24376
TIME(s)         COMM           PID    NODE ZONE         ORDER MODE      LAT(ms)           STATUS
101.364115000   zsh            24376  0    ZONE_DMA     -1    SYNC        0.025         complete
101.364555000   zsh            24376  0    ZONE_DMA32   -1    SYNC        3.925         complete
^C

This shows the zsh process allocs pages, and compact zone events happening,
and the delays are not affected much.

A maximum tracing duration can be set with the -d option. For example, to trace
for 2 seconds:

# ./compactsnoop -d 2
COMM           PID    NODE ZONE         ORDER MODE       LAT(ms)           STATUS
zsh            26385  0    ZONE_DMA     -1    SYNC      0.025444         complete
^C

The -e option prints out extra columns

# ./compactsnoop -e
COMM           PID    NODE ZONE         ORDER MODE    FRAGIDX  MIN      LOW      HIGH     FREE       LAT(ms)           STATUS
summ           28276  1    ZONE_NORMAL  3     ASYNC   0.728    11284    14105    16926    14193         3.58          partial
summ           28276  0    ZONE_NORMAL  2     ASYNC   -1.000   11043    13803    16564    14479          0.0         complete
summ           28276  1    ZONE_NORMAL  2     ASYNC   -1.000   11284    14105    16926    14785        0.019         complete
summ           28276  0    ZONE_NORMAL  2     ASYNC   -1.000   11043    13803    16564    15199        0.006          partial
summ           28276  1    ZONE_NORMAL  2     ASYNC   -1.000   11284    14105    16926    17360        0.030         complete
summ           28276  0    ZONE_NORMAL  2     ASYNC   -1.000   11043    13803    16564    15443        0.024         complete
summ           28276  1    ZONE_NORMAL  2     ASYNC   -1.000   11284    14105    16926    15634        0.018         complete
summ           28276  1    ZONE_NORMAL  3     ASYNC   0.832    11284    14105    16926    15301        0.006          partial
summ           28276  0    ZONE_NORMAL  2     ASYNC   -1.000   11043    13803    16564    14774        0.005          partial
summ           28276  1    ZONE_NORMAL  3     ASYNC   0.733    11284    14105    16926    19888        0.012          partial
^C

The FRAGIDX is short for fragmentation index, which only makes sense if an
allocation of a requested size would fail. If that is true, the fragmentation
index indicates whether external fragmentation or a lack of memory was the
problem. The value can be used to determine if page reclaim or compaction
should be used.

Index is between 0 and 1 so return within 3 decimal places

0 => allocation would fail due to lack of memory
1 => allocation would fail due to fragmentation

We can see the whole buddy's fragmentation index from /sys/kernel/debug/extfrag/extfrag_index

The MIN/LOW/HIGH shows the watermarks of the zone, which can also get from
/proc/zoneinfo, and FREE means nr_free_pages (can be found in /proc/zoneinfo too).


The -K option prints out kernel stack

# ./compactsnoop -K -e

summ           28276  0    ZONE_NORMAL  3     ASYNC   0.528    11043    13803    16564    22654       13.258          partial
               kretprobe_trampoline+0x0
               try_to_compact_pages+0x121
               __alloc_pages_direct_compact+0xac
               __alloc_pages_slowpath+0x3e9
               __alloc_pages_nodemask+0x404
               alloc_pages_current+0x98
               new_slab+0x2c5
               ___slab_alloc+0x3ac
               __slab_alloc+0x40
               kmem_cache_alloc_node+0x8b
               copy_process+0x18e
               do_fork+0x91
               sys_clone+0x16
               stub_clone+0x44

summ           28276  1    ZONE_NORMAL  3     ASYNC   -1.000   11284    14105    16926    22074        0.008          partial
               kretprobe_trampoline+0x0
               try_to_compact_pages+0x121
               __alloc_pages_direct_compact+0xac
               __alloc_pages_slowpath+0x3e9
               __alloc_pages_nodemask+0x404
               alloc_pages_current+0x98
               new_slab+0x2c5
               ___slab_alloc+0x3ac
               __slab_alloc+0x40
               kmem_cache_alloc_node+0x8b
               copy_process+0x18e
               do_fork+0x91
               sys_clone+0x16
               stub_clone+0x44

summ           28276  0    ZONE_NORMAL  3     ASYNC   0.527    11043    13803    16564    25653        9.812          partial
               kretprobe_trampoline+0x0
               try_to_compact_pages+0x121
               __alloc_pages_direct_compact+0xac
               __alloc_pages_slowpath+0x3e9
               __alloc_pages_nodemask+0x404
               alloc_pages_current+0x98
               new_slab+0x2c5
               ___slab_alloc+0x3ac
               __slab_alloc+0x40
               kmem_cache_alloc_node+0x8b
               copy_process+0x18e
               do_fork+0x91
               sys_clone+0x16
               stub_clone+0x44

# ./compactsnoop -h
usage: compactsnoop.py [-h] [-T] [-p PID] [-d DURATION] [-K] [-e]

Trace compact zone

optional arguments:
  -h, --help            show this help message and exit
  -T, --timestamp       include timestamp on output
  -p PID, --pid PID     trace this PID only
  -d DURATION, --duration DURATION
                        total duration of trace in seconds
  -K, --kernel-stack    output kernel stack trace
  -e, --extended_fields
                        show system memory state

examples:
    ./compactsnoop          # trace all compact stall
    ./compactsnoop -T       # include timestamps
    ./compactsnoop -d 10    # trace for 10 seconds only
    ./compactsnoop -K       # output kernel stack trace
    ./compactsnoop -e       # show extended fields