asmhatre/java-core-examples
1
0
Fork 0
Code Activity

nio2: NIO.2 file API companion code (streaming 5 GB at -Xmx64m, handle leaks, WatchService, mmap)

Browse Source 
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01KqJyCidz3ZgRyHABv2GVJh
This commit is contained in:
asmhatre
2026-09-30 19:14:31 +00:00
co-authored by Claude Sonnet 5.5
parent f4b732799d
commit 1ac2f6077a
30 changed files with 1072 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
nio2/output/01-environment.txt
+18
View File
@@ -0,0 +1,18 @@
$ uname -sr
Linux 6.18.44-fc-v50
$ nproc
2
$ java -version
openjdk version "25.0.4.1" 2026-08-18 LTS
OpenJDK Runtime Environment Temurin-25.0.4.1+1 (build 25.0.4.1+1-LTS)
OpenJDK 64-Bit Server VM Temurin-25.0.4.1+1 (build 25.0.4.1+1-LTS, mixed mode, sharing)
$ ulimit -n
20000
$ cat /proc/sys/fs/inotify/max_queued_events
16384
$ df -h (data directory, before)
Filesystem Size Used Avail Use% Mounted on
/dev/vda 252G 13G 29G 31% /
$ df -h (data directory, after deleting is done by the exit trap; shown here before)
Filesystem Size Used Avail Use% Mounted on
/dev/vda 252G 18G 25G 43% /
nio2/output/02-generate.txt
+8
View File
@@ -0,0 +1,8 @@
generated big.log: 5,000,000,069 bytes, 34,120,522 lines, 681,844 ERROR lines in 13.7 s
generated medium.log: 200,000,016 bytes, 1,378,768 lines, 27,530 ERROR lines in 1.4 s
$ ls -l (generated files)
5000000069 big.log
200000016 medium.log
$ df -h (data directory, with both files on disk)
Filesystem Size Used Avail Use% Mounted on
/dev/vda 252G 18G 25G 43% /
nio2/output/03-stream-lines.txt
+17
View File
@@ -0,0 +1,17 @@
$ java -Xmx64m -Xlog:gc StreamLines lines big.log
Files.lines over big.log (5,000,000,069 bytes)
lines=34,120,522 errors=681,844 services=20
elapsed 9.4 s
heap: -Xmx=64 MiB, peak used (sampled every 20 ms)=38 MiB, GC collections=222, GC time=200 ms
GC log: 223 lines, 222 young pauses, 0 full GCs
first lines of the GC log:
[0.004s][info][gc] Using G1
[0.321s][info][gc] GC(0) Pause Young (Normal) (G1 Evacuation Pause) 15M->1M(64M) 7.040ms
[0.440s][info][gc] GC(1) Pause Young (Normal) (G1 Evacuation Pause) 28M->1M(64M) 1.218ms
[0.497s][info][gc] GC(2) Pause Young (Normal) (G1 Evacuation Pause) 38M->1M(64M) 1.372ms
last lines of the GC log:
[9.358s][info][gc] GC(219) Pause Young (Normal) (G1 Evacuation Pause) 38M->1M(64M) 0.696ms
[9.396s][info][gc] GC(220) Pause Young (Normal) (G1 Evacuation Pause) 38M->1M(64M) 0.466ms
[9.430s][info][gc] GC(221) Pause Young (Normal) (G1 Evacuation Pause) 38M->1M(64M) 0.497ms
largest 'Pause Young' (ms): 7.040
highest heap-after-GC seen in the log: 1M of 64M
nio2/output/04-stream-reader.txt
+17
View File
@@ -0,0 +1,17 @@
$ java -Xmx64m -Xlog:gc StreamLines reader big.log
Files.newBufferedReader over big.log (5,000,000,069 bytes)
lines=34,120,522 errors=681,844 services=20
elapsed 9.3 s
heap: -Xmx=64 MiB, peak used (sampled every 20 ms)=38 MiB, GC collections=222, GC time=212 ms
GC log: 223 lines, 222 young pauses, 0 full GCs
first lines of the GC log:
[0.005s][info][gc] Using G1
[0.273s][info][gc] GC(0) Pause Young (Normal) (G1 Evacuation Pause) 15M->1M(64M) 2.473ms
[0.349s][info][gc] GC(1) Pause Young (Normal) (G1 Evacuation Pause) 28M->1M(64M) 1.211ms
[0.401s][info][gc] GC(2) Pause Young (Normal) (G1 Evacuation Pause) 38M->1M(64M) 1.173ms
last lines of the GC log:
[9.286s][info][gc] GC(219) Pause Young (Normal) (G1 Evacuation Pause) 38M->1M(64M) 0.515ms
[9.319s][info][gc] GC(220) Pause Young (Normal) (G1 Evacuation Pause) 38M->1M(64M) 0.427ms
[9.353s][info][gc] GC(221) Pause Young (Normal) (G1 Evacuation Pause) 38M->1M(64M) 0.469ms
largest 'Pause Young' (ms): 6.615
highest heap-after-GC seen in the log: 1M of 64M
nio2/output/05-read-all-fail.txt
+22
View File
@@ -0,0 +1,22 @@
$ java -Xmx64m ReadAllFail readAllLines medium.log
OutOfMemoryError: Java heap space
$ java -Xmx64m ReadAllFail readString medium.log
OutOfMemoryError: Java heap space
$ java -Xmx64m ReadAllFail readAllBytes medium.log
OutOfMemoryError: Java heap space
$ java -Xmx64m ReadAllFail readAllLines big.log
OutOfMemoryError: Java heap space
$ java -Xmx64m ReadAllFail readString big.log
OutOfMemoryError: Required array size too large
$ java -Xmx64m ReadAllFail readAllBytes big.log
OutOfMemoryError: Required array size too large
How much heap does the 200 MB file need? (readAllLines, then readString)
readAllLines -Xmx256m OutOfMemoryError: Java heap space
readString -Xmx256m survived: 200000016 chars
readAllLines -Xmx512m survived: 1378768 lines
readString -Xmx512m survived: 200000016 chars
readAllLines -Xmx1g survived: 1378768 lines
readString -Xmx1g survived: 200000016 chars
readAllLines -Xmx2g survived: 1378768 lines
readString -Xmx2g survived: 200000016 chars
nio2/output/06-handle-leak.txt
+11
View File
@@ -0,0 +1,11 @@
open fds at start: 8
after 50 fully consumed, unclosed Files.walk: +0 fds
after 50 unclosed Files.lines + findFirst(): +50 fds
after closing them: +0 fds
after 50 try-with-resources Files.lines: +0 fds
after 50 unclosed Files.walk + findFirst(): +300 fds
after closing them: +0 fds
after 50 unreferenced, unclosed Files.lines: +50 fds
after System.gc(): +0 fds
after 50 unreferenced, unclosed Files.walk: +300 fds
after System.gc(): +300 fds
nio2/output/07-handle-exhaustion.txt
+3
View File
@@ -0,0 +1,3 @@
$ ulimit -n 64; java HandleExhaustion
failed on call 10 after 9 leaked walks
java.io.UncheckedIOException: java.nio.file.FileSystemException: <tmp>/tree/d2/sub: Too many open files
nio2/output/08-walk-find.txt
+9
View File
@@ -0,0 +1,9 @@
Files.list (one level): [.git, README.md, src]
Files.walk (everything): [, .git, .git/objects, .git/objects/blob.bin, README.md, src, src/main, src/main/App.java, src/test, src/test/AppTest.java]
Files.walk maxDepth=1: [, .git, README.md, src]
Files.find *.java: [src/main/App.java, src/test/AppTest.java]
Files.find size > 1000: [src/test/AppTest.java]
walkFileTree skipping .git: [README.md, src/main/App.java, src/test/AppTest.java]
DirectoryStream glob: [App.java]
walk with a symlink loop, default: 11 entries, no error
walk with FOLLOW_LINKS: FileSystemLoopException
nio2/output/09-lines-charset.txt
+4
View File
@@ -0,0 +1,4 @@
lines processed before failure: 0
UncheckedIOException caused by java.nio.charset.MalformedInputException: Input length = 1
with ISO_8859_1: 3 lines, no error
readAllLines throws checked MalformedInputException
nio2/output/10-watch.txt
+19
View File
@@ -0,0 +1,19 @@
1. create a file, write it twice, delete it (events polled afterwards):
ENTRY_CREATE count=1 context=a.txt
ENTRY_MODIFY count=2 context=a.txt
ENTRY_DELETE count=1 context=a.txt
2. Files.writeString of 100 KiB to a new file (one call):
ENTRY_CREATE count=1 context=b.txt
ENTRY_MODIFY count=1 context=b.txt
3. 50 appends to one file with no polling in between, then poll:
{(batches)=1, ENTRY_MODIFY=9}
4. Files.move (atomic rename) of g.txt to h.txt:
ENTRY_DELETE count=1 context=g.txt
ENTRY_CREATE count=1 context=h.txt
5. a file created inside a NEW subdirectory (subdirectory not registered):
ENTRY_CREATE count=1 context=sub
6. burst: create N empty files in a fresh directory before polling once (jdk.nio.file.WatchService.maxEventsPerPoll=default):
N=100 delivered 100 events: ENTRY_CREATE=100 OVERFLOW(count)=0, files on disk=100
N=512 delivered 512 events: ENTRY_CREATE=512 OVERFLOW(count)=0, files on disk=512
N=513 delivered 1 events: ENTRY_CREATE=0 OVERFLOW(count)=1, files on disk=513
N=5000 delivered 1 events: ENTRY_CREATE=0 OVERFLOW(count)=4488, files on disk=5000
nio2/output/11-watch-tuned-and-repeat.txt
+13
View File
@@ -0,0 +1,13 @@
$ java -Djdk.nio.file.WatchService.maxEventsPerPoll=10000 WatchDemo (scenario 6 only)
6. burst: create N empty files in a fresh directory before polling once (jdk.nio.file.WatchService.maxEventsPerPoll=10000):
N=100 delivered 100 events: ENTRY_CREATE=100 OVERFLOW(count)=0, files on disk=100
N=512 delivered 512 events: ENTRY_CREATE=512 OVERFLOW(count)=0, files on disk=512
N=513 delivered 513 events: ENTRY_CREATE=513 OVERFLOW(count)=0, files on disk=513
N=5000 delivered 5000 events: ENTRY_CREATE=5000 OVERFLOW(count)=0, files on disk=5000
Scenario 3 (50 appends, no polling) over five more runs:
{(batches)=1, ENTRY_MODIFY=21}
{(batches)=1, ENTRY_MODIFY=20}
{(batches)=1, ENTRY_MODIFY=2}
{(batches)=1, ENTRY_MODIFY=1}
{(batches)=1, ENTRY_MODIFY=19}
nio2/output/12-watchkey-source-excerpt.txt
+54
View File
@@ -0,0 +1,54 @@
final void signalEvent(WatchEvent.Kind<?> kind, Object context) {
boolean isModify = (kind == StandardWatchEventKinds.ENTRY_MODIFY);
synchronized (this) {
int size = events.size();
if (size > 0) {
// if the previous event is an OVERFLOW event or this is a
// repeated event then we simply increment the counter
WatchEvent<?> prev = events.get(size-1);
if ((prev.kind() == StandardWatchEventKinds.OVERFLOW) ||
((kind == prev.kind() &&
Objects.equals(context, prev.context()))))
{
((Event<?>)prev).increment();
return;
}
// if this is a modify event and the last entry for the context
// is a modify event then we simply increment the count
if (!lastModifyEvents.isEmpty()) {
if (isModify) {
WatchEvent<?> ev = lastModifyEvents.get(context);
if (ev != null) {
assert ev.kind() == StandardWatchEventKinds.ENTRY_MODIFY;
((Event<?>)ev).increment();
return;
}
} else {
// not a modify event so remove from the map as the
// last event will no longer be a modify event.
lastModifyEvents.remove(context);
}
}
// if the list has reached the limit then drop pending events
// and queue an OVERFLOW event
if (size >= MAX_EVENT_LIST_SIZE) {
kind = StandardWatchEventKinds.OVERFLOW;
isModify = false;
context = null;
}
}
// non-repeated event
Event<Object> ev =
new Event<>((WatchEvent.Kind<Object>)kind, context);
if (isModify) {
lastModifyEvents.put(context, ev);
} else if (kind == StandardWatchEventKinds.OVERFLOW) {
// drop all pending events
events.clear();
lastModifyEvents.clear();
}
events.add(ev);
signal();
nio2/output/13-mapped.txt
+7
View File
@@ -0,0 +1,7 @@
$ java -Xmx64m MappedDemo big.log
file big.log: 5,000,000,069 bytes, -Xmx=64 MiB, RSS at start=40 MiB
classic map of the whole file: IllegalArgumentException: Size exceeds Integer.MAX_VALUE
Arena + MemorySegment: 34,120,522 newlines in 5.1 s; RSS after the Arena was closed=51 MiB
classic 1 GiB windows: 34,120,522 newlines in 7.8 s; RSS right after=4821 MiB
heap: -Xmx=64 MiB, peak used (sampled every 20 ms)=4 MiB, GC collections=0, GC time=0 ms
RSS one second after System.gc()=52 MiB
nio2/output/14-tests.txt
+4
View File
@@ -0,0 +1,4 @@
-------------------------------------------------------------------------------
Test set: com.ankurm.nio2.Nio2BehaviourTest
-------------------------------------------------------------------------------
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.755 s -- in com.ankurm.nio2.Nio2BehaviourTest