max heap        : 512 MB
target live set : 281 MB (55% of heap)
threads         : 12, duration: 20 s

filling live set ...
live set filled: 4,505 chunks x 64 KB
now allocating hard for 20 s -- expect stalls

allocation phase done

========================================================================
JFR summary  (C:\Users\Ankur\allocation-stalls.jfr)
========================================================================
young collections : 1,722
old collections   : 219
page allocations  : 39,803
ALLOCATION STALLS : 32,442

total time threads spent stalled : 140,558 ms
mean stall                       : 4.33 ms
longest stall                    : 50.33 ms

stalls by ZGC page type
  Small    count=32,442   total=140,558 ms   mean=4.33 ms

ten longest stalls
  duration    page     size       thread
     50.33 ms Small    2097152    allocator-10
     48.98 ms Small    2097152    allocator-5
     48.96 ms Small    2097152    allocator-6
     48.84 ms Small    2097152    allocator-7
     48.79 ms Small    2097152    allocator-1
     47.89 ms Small    2097152    allocator-0
     47.33 ms Small    2097152    allocator-10
     47.12 ms Small    2097152    allocator-3
     46.96 ms Small    2097152    allocator-5
     46.71 ms Small    2097152    allocator-1

Compare against your pause-time chart: none of the above is a pause.
Inspect the raw events, including the stack that was stalled, with:
  jfr print --events ZAllocationStall allocation-stalls.jfr