max heap        : 512 MB
target live set : 281 MB (55% of heap)
threads         : 12, duration: 20 s

filling live set ...
live set filled: 4,505 chunks x 64 KB
now allocating hard for 20 s -- expect stalls

allocation phase done

========================================================================
JFR summary  (C:\Users\Ankur\allocation-stalls.jfr)
========================================================================
young collections : 1,771
old collections   : 217
page allocations  : 39,796
ALLOCATION STALLS : 32,200

total time threads spent stalled : 139,439 ms
mean stall                       : 4.33 ms
longest stall                    : 48.32 ms

stalls by ZGC page type
  Small    count=32,200   total=139,439 ms   mean=4.33 ms

ten longest stalls
  duration    page     size       thread
     48.32 ms Small    2097152    allocator-3
     47.99 ms Small    2097152    allocator-10
     46.39 ms Small    2097152    allocator-1
     46.23 ms Small    2097152    allocator-0
     46.19 ms Small    2097152    allocator-7
     46.18 ms Small    2097152    allocator-5
     45.11 ms Small    2097152    allocator-6
     44.85 ms Small    2097152    allocator-7
     44.68 ms Small    2097152    allocator-10
     44.49 ms Small    2097152    allocator-0

Compare against your pause-time chart: none of the above is a pause.
Inspect the raw events, including the stack that was stalled, with:
  jfr print --events ZAllocationStall allocation-stalls.jfr