max heap        : 512 MB
target live set : 281 MB (55% of heap)
threads         : 12, duration: 20 s

filling live set ...
live set filled: 4,505 chunks x 64 KB
now allocating hard for 20 s -- expect stalls

allocation phase done

========================================================================
JFR summary  (C:\Users\Ankur\allocation-stalls.jfr)
========================================================================
young collections : 1,778
old collections   : 217
page allocations  : 39,685
ALLOCATION STALLS : 31,967

total time threads spent stalled : 138,521 ms
mean stall                       : 4.33 ms
longest stall                    : 51.41 ms

stalls by ZGC page type
  Small    count=31,967   total=138,521 ms   mean=4.33 ms

ten longest stalls
  duration    page     size       thread
     51.41 ms Small    2097152    allocator-10
     50.18 ms Small    2097152    allocator-0
     48.12 ms Small    2097152    allocator-5
     47.93 ms Small    2097152    allocator-1
     46.90 ms Small    2097152    allocator-1
     46.14 ms Small    2097152    allocator-11
     46.14 ms Small    2097152    allocator-11
     46.08 ms Small    2097152    allocator-6
     46.02 ms Small    2097152    allocator-4
     45.68 ms Small    2097152    allocator-9

Compare against your pause-time chart: none of the above is a pause.
Inspect the raw events, including the stack that was stalled, with:
  jfr print --events ZAllocationStall allocation-stalls.jfr