Java NIO.2 File API: Files, Path, WatchService, and Streaming Large Files Without OOM
Stream a 5 GB file at a 64 MB heap with GC evidence, then see the failure modes: readAllLines/readString OOM, unclosed Files.lines and Files.walk leaking file handles, WatchService event folding and the 512-event overflow cliff on Linux, and memory-mapped files. All measured on JDK 25 with a runnable companion module.
Picture a nightly export job that reads its input with Files.readAllLines. On a developer laptop the input is a small sample and the job takes a second. In production the input is several gigabytes and the job dies with OutOfMemoryError: Java heap space before it does any work. Nothing in the code is wrong in the usual sense; the method does exactly what its name says, and on a big file that is the bug. (The Javadoc for readAllLines and readAllBytes both say so: “It is not intended for reading in large files.”)
This article starts from that bug and works outward through Java’s file API, NIO.2 (the java.nio.file package: Path, Files, WatchService). You will see the same 5 GB file processed at a 64 MB heap, then every way the obvious code goes wrong: whole-file reads that run out of memory, streams that quietly hold a file handle, directory walks that leak handles, a WatchService that drops events, and memory-mapped files that are not as free as they look. If you only know BufferedReader, start with Java BufferedReader Tutorial; this page picks up where that one stops.
Versions.JDK 25.0.4.1+1 (Temurin, LTS), JUnit Jupiter 5.11.0, Linux kernel 6.18 on a 2-vCPU virtual machine that other jobs were also using, so timings are indicative only; counts, exceptions and GC figures are not. The large file is 5,000,000,069 bytes (34,120,522 lines): the full 5 GB fit on the sandbox disk, so nothing was scaled down. Every code block and every line of output below comes from the nio2 module of the java-core-examples repository, regenerated by one script. I tested Linux only; the WatchService and file-handle results are Linux results, and I say so where it matters.
Reading a whole file into memory works right up to the day it does not
Java gives you three one-line ways to read a file completely: Files.readAllLines(path) returns a List<String>, Files.readString(path) returns one big String, and Files.readAllBytes(path) returns a byte[]. They are wonderful for configuration files and test fixtures. They all have the same property: the entire file lives on the heap before your code sees the first byte.
The demo runs each of them with -Xmx64m (a 64 MiB heap limit) against two generated files: a 200 MB one and the 5 GB one. The file is synthetic log lines, produced by LogFile.java with a fixed seed so every run is identical. The reading code is the smallest thing that can fail:
try {
Object r = switch (a[0]) {
case "readAllLines" -> Files.readAllLines(p);
case "readString" -> Files.readString(p);
case "readAllBytes" -> Files.readAllBytes(p);
default -> throw new IllegalArgumentException(a[0]);
};
System.out.println("survived: " + (r instanceof java.util.List<?> l ? l.size() + " lines"
: r instanceof String s ? s.length() + " chars" : ((byte[]) r).length + " bytes"));
} catch (OutOfMemoryError e) {
System.out.println("OutOfMemoryError: " + e.getMessage());
}
$ java -Xmx64m ReadAllFail readAllLines medium.log
OutOfMemoryError: Java heap space
$ java -Xmx64m ReadAllFail readString medium.log
OutOfMemoryError: Java heap space
$ java -Xmx64m ReadAllFail readAllBytes medium.log
OutOfMemoryError: Java heap space
$ java -Xmx64m ReadAllFail readAllLines big.log
OutOfMemoryError: Java heap space
$ java -Xmx64m ReadAllFail readString big.log
OutOfMemoryError: Required array size too large
$ java -Xmx64m ReadAllFail readAllBytes big.log
OutOfMemoryError: Required array size too large
All three fail on the 200 MB file with Java heap space, as you would expect. The 5 GB file shows something more useful: readAllLines still fails with heap exhaustion, but readString and readAllBytes fail with a different message, Required array size too large. That one has nothing to do with -Xmx: a Java array (and therefore a byte[] or a String) cannot hold more than about 2 billion elements, so a file of 5 GB cannot be read this way on any heap. Raising -Xmx fixes the first failure and cannot fix the second.
The diagram is the whole idea of the article. The right-hand numbers are measured, not assumed; the next section shows where they come from.
How much heap does a whole-file read actually need? The second half of the same transcript tries the 200 MB file (190.7 MiB) at four heap sizes:
How much heap does the 200 MB file need? (readAllLines, then readString)
readAllLines -Xmx256m OutOfMemoryError: Java heap space
readString -Xmx256m survived: 200000016 chars
readAllLines -Xmx512m survived: 1378768 lines
readString -Xmx512m survived: 200000016 chars
readAllLines -Xmx1g survived: 1378768 lines
readString -Xmx1g survived: 200000016 chars
readAllLines -Xmx2g survived: 1378768 lines
readString -Xmx2g survived: 200000016 chars
readAllLines failed at 256 MiB and succeeded at 512 MiB, so for this file it needed somewhere between 1.3 and 2.7 times the file size; readString succeeded already at 256 MiB. I did not bisect further, so treat the range, not a factor, as the finding. The practical rule is the same either way: heap needed grows with file size, and you do not control the file size.
Going deeper
Files.readAllLines and Files.readString in the Files Javadoc
Stream the file: a 5 GB file at a 64 MB heap, with the GC log to prove it
The fix is to never hold more than a line at a time. NIO.2 offers two idiomatic ways. Files.lines(path) returns a lazy Stream<String> that reads the file as your pipeline pulls lines from it. Files.newBufferedReader(path) returns a BufferedReader you loop over yourself; it is the same reader the BufferedReader tutorial covers, just obtained through Path and with UTF-8 as the default charset. Both do the same job here: count lines, count lines that start with ERROR, and count lines per service name.
/** Files.lines: lazy Stream<String>; MUST be closed (it owns a file handle). */
public static Result withFilesLines(Path p) throws IOException {
long[] c = new long[2];
Map<String, Long> per = new TreeMap<>();
try (Stream<String> s = Files.lines(p)) {
s.forEach(l -> tally(l, c, per));
}
return new Result(c[0], c[1], per);
}
/** Files.newBufferedReader: the same thing with a plain loop. */
public static Result withBufferedReader(Path p) throws IOException {
long[] c = new long[2];
Map<String, Long> per = new TreeMap<>();
try (BufferedReader r = Files.newBufferedReader(p)) {
for (String l = r.readLine(); l != null; l = r.readLine()) tally(l, c, per);
}
return new Result(c[0], c[1], per);
}
private static void tally(String line, long[] c, Map<String, Long> per) {
c[0]++;
if (line.startsWith("ERROR", 20)) c[1]++;
int i = line.indexOf("service-");
per.merge(line.substring(i, line.indexOf(' ', i)), 1L, Long::sum);
}
Run with -Xmx64m -Xlog:gc, the 5 GB file through Files.lines (03-stream-lines.txt):
$ java -Xmx64m -Xlog:gc StreamLines lines big.log
Files.lines over big.log (5,000,000,069 bytes)
lines=34,120,522 errors=681,844 services=20
elapsed 9.4 s
heap: -Xmx=64 MiB, peak used (sampled every 20 ms)=38 MiB, GC collections=222, GC time=200 ms
GC log: 223 lines, 222 young pauses, 0 full GCs
first lines of the GC log:
[0.004s][info][gc] Using G1
[0.321s][info][gc] GC(0) Pause Young (Normal) (G1 Evacuation Pause) 15M->1M(64M) 7.040ms
[0.440s][info][gc] GC(1) Pause Young (Normal) (G1 Evacuation Pause) 28M->1M(64M) 1.218ms
[0.497s][info][gc] GC(2) Pause Young (Normal) (G1 Evacuation Pause) 38M->1M(64M) 1.372ms
last lines of the GC log:
[9.358s][info][gc] GC(219) Pause Young (Normal) (G1 Evacuation Pause) 38M->1M(64M) 0.696ms
[9.396s][info][gc] GC(220) Pause Young (Normal) (G1 Evacuation Pause) 38M->1M(64M) 0.466ms
[9.430s][info][gc] GC(221) Pause Young (Normal) (G1 Evacuation Pause) 38M->1M(64M) 0.497ms
largest 'Pause Young' (ms): 7.040
highest heap-after-GC seen in the log: 1M of 64M
Here is what that transcript says. The JVM chose G1 and made 222 young-generation collections and zero full collections while reading 34,120,522 lines. After every collection the live heap was 1 MiB of the 64 MiB limit; the 38 MiB peak is just young-generation garbage piling up between collections, which is what a healthy streaming job looks like. The BufferedReader version (04-stream-reader.txt) produced the same counts and the same 222 collections:
$ java -Xmx64m -Xlog:gc StreamLines reader big.log
Files.newBufferedReader over big.log (5,000,000,069 bytes)
lines=34,120,522 errors=681,844 services=20
elapsed 9.3 s
heap: -Xmx=64 MiB, peak used (sampled every 20 ms)=38 MiB, GC collections=222, GC time=212 ms
GC log: 223 lines, 222 young pauses, 0 full GCs
So the choice between the two is about style, not memory. A stream is convenient when you want filter/map/collect; a reader loop is convenient when you need to keep state across lines or to break out cleanly. Elapsed times in both transcripts are close to each other, but this was a shared 2-vCPU machine and I did not repeat the runs enough to rank them, so I am not claiming one is faster.
What ‘constant heap’ does and does not mean. The heap stayed flat because the processing is constant-memory: a few counters and a 20-entry map. If your per-line work collects lines into a list, a Collectors.toList(), or a map keyed by something unbounded, you have rebuilt readAllLines with extra steps and the OOM comes back. Also note the measurement: peak used heap is sampled by a 20 ms daemon thread in HeapProbe.java, so it can miss a short spike; the GC log, which records every collection, is the stronger evidence.
Reference: how the 5 GB file was made and what was deleted afterwardsdf showed 29 GB free before the run (01-environment.txt), so the full 5 GB went ahead. LogFile.java writes whole lines until the target size is reached, so the file ended at 5,000,000,069 bytes rather than exactly 5,000,000,000. The generator and the 200 MB file are in 02-generate.txt:
generated big.log: 5,000,000,069 bytes, 34,120,522 lines, 681,844 ERROR lines in 13.7 s
generated medium.log: 200,000,016 bytes, 1,378,768 lines, 27,530 ERROR lines in 1.4 s
The script deletes both files on exit (a shell trap), and BIG_BYTES lets you run with a smaller file if your disk is tight. The scripts and tests never need the 5 GB file to be present afterwards.
Going deeper
A stream from Files.lines owns an open file, so close it
Look again at the try (Stream<String> s = Files.lines(p)) line in the streaming code above. The try-with-resources is not decoration. A Stream does not normally need closing, but this one wraps an open file, and the operating system gives every open file a small integer (a file descriptor) from a limited table. Closing the stream closes the file. The Javadoc says it in an API note: the method “must be used within a try-with-resources statement or similar control structure to ensure that the stream’s open file is closed promptly”.
To see that in numbers rather than trust the note, HandleLeakDemo.java counts the entries in /proc/self/fd (Linux’s list of this process’s open descriptors) around 50 uses of each pattern:
From HandleLeakDemo.java:
// 1. Files.lines, opened and read partially, never closed. References are kept (as a field or cache would).
List<Stream<String>> keep = new ArrayList<>();
for (int i = 0; i < N; i++) { Stream<String> s = Files.lines(file); s.findFirst(); keep.add(s); }
System.out.printf("after %d unclosed Files.lines + findFirst(): +%d fds%n", N, openFds() - base);
keep.forEach(Stream::close);
System.out.printf("after closing them: +%d fds%n", openFds() - base);
// 2. Files.lines in try-with-resources, same work.
for (int i = 0; i < N; i++) { try (Stream<String> s = Files.lines(file)) { s.findFirst(); } }
System.out.printf("after %d try-with-resources Files.lines: +%d fds%n", N, openFds() - base);
open fds at start: 8
...
after 50 unclosed Files.lines + findFirst(): +50 fds
after closing them: +0 fds
after 50 try-with-resources Files.lines: +0 fds
Read the first lines of the transcript against the diagram: 50 forgotten Files.lines streams cost 50 descriptors; closing them brings the count back to the baseline; 50 streams in try-with-resources cost nothing. The same file (06-handle-leak.txt) also contains a result worth being honest about:
after 50 unreferenced, unclosed Files.lines: +50 fds
after System.gc(): +0 fds
When 50 unclosed Files.lines streams were not referenced anywhere, a System.gc() brought the descriptor count back to the baseline. The JDK evidently cleans up an abandoned file stream once the collector finds it (I observed this; I did not trace which JDK object does it). That is not a safety net: nothing tells you when a collection will run, and a heap that barely allocates (like the streaming job above, at 1 MiB live) may go a long time between collections while descriptors pile up. Close what you open.
Files.lines is lazy in one more way that catches people out: decoding happens as lines are pulled, so a bad byte fails in the middle of the pipeline, as an unchecked exception:
From LinesCharsetDemo.java, with output in 09-lines-charset.txt:
try (Stream<String> s = Files.lines(p)) {
s.forEach(l -> seen[0]++);
} catch (UncheckedIOException e) {
System.out.println("lines processed before failure: " + seen[0]);
System.out.println(e.getClass().getSimpleName() + " caused by " + e.getCause());
}
try (Stream<String> s = Files.lines(p, StandardCharsets.ISO_8859_1)) {
System.out.println("with ISO_8859_1: " + s.count() + " lines, no error");
}
lines processed before failure: 0
UncheckedIOException caused by java.nio.charset.MalformedInputException: Input length = 1
with ISO_8859_1: 3 lines, no error
readAllLines throws checked MalformedInputException
Files.lines wraps the MalformedInputException in an UncheckedIOException, whereas Files.readAllLines throws the checked one (third line of the output). The default charset for both is UTF-8, so a Latin-1 file with an accented letter fails on the first such byte unless you pass StandardCharsets.ISO_8859_1. (The bad byte is on the third line, yet the counter printed 0: the exception arrived before any line reached my code. I did not investigate why, so do not count on partial progress before a decoding failure.)
Going deeper
Files.walk and Files.find are streams of directories: close them too, and know what early exit leaves open
NIO.2 has four ways to look inside a directory, and picking the right one matters more than it looks. The demo builds a tiny project tree and runs each one (WalkFindDemo.java):
Files.list is one level. Files.walk visits everything below, including the starting directory itself (the empty string in the output is the root). Files.find is walk plus a predicate that also receives each file’s attributes, and the Javadoc says it “may be more efficient” than filtering a walk because it avoids retrieving the attributes twice. Two more exist for special cases: Files.walkFileTree with a FileVisitor, which can skip a whole subtree such as .git, and Files.newDirectoryStream with a glob, which filters names on a single level.
Pruning and globbing, from WalkFindDemo.java (output: 08-walk-find.txt):
All of the stream-returning ones (list, walk, find) have the same ownership rule as Files.lines; the Javadoc for walk says the returned stream “contains references to one or more open directories” and that “the directories are closed by closing the stream”. The demo helper closes each one with try (s). What happens if you do not? The same descriptor counting, this time on Files.walk:
The walk cases, from HandleLeakDemo.java (output: 06-handle-leak.txt):
// 0. Files.walk fully consumed but not closed, run first so later leaks do not hide it.
for (int i = 0; i < N; i++) { Files.walk(tree).count(); }
System.out.printf("after %d fully consumed, unclosed Files.walk: +%d fds%n", N, openFds() - base);
// 3. Files.walk, findFirst(), never closed.
List<Stream<Path>> walks = new ArrayList<>();
for (int i = 0; i < N; i++) { Stream<Path> s = Files.walk(tree); s.skip(3).findFirst(); walks.add(s); }
System.out.printf("after %d unclosed Files.walk + findFirst(): +%d fds%n", N, openFds() - base);
walks.forEach(Stream::close);
System.out.printf("after closing them: +%d fds%n", openFds() - base);
after 50 fully consumed, unclosed Files.walk: +0 fds
...
after 50 unclosed Files.walk + findFirst(): +300 fds
after closing them: +0 fds
...
after 50 unreferenced, unclosed Files.walk: +300 fds
after System.gc(): +300 fds
Three things are in those lines. A walk that is fully consumed (count() read every element) left no descriptors behind: +0 after 50 of them. A walk that stops early (here skip(3).findFirst(), which is how people write “give me the first match”) kept directories open: +300 for 50 calls, six descriptors each on this small tree. And unlike the Files.lines case, System.gc() did not reclaim them: the count stayed at +300. So Files.walk(dir).findFirst() without try-with-resources is a real leak, not a theoretical one. I did not check why six per call, and would not assume a fixed number for other trees.
What does a leak like that turn into? HandleExhaustion.java repeats the early-exit walk under ulimit -n 64 until the process runs out:
$ ulimit -n 64; java HandleExhaustion
failed on call 10 after 9 leaked walks
java.io.UncheckedIOException: java.nio.file.FileSystemException: <tmp>/tree/d2/sub: Too many open files
The fingerprint of a descriptor leak.FileSystemException: ... Too many open files, thrown from an UncheckedIOException, on a call to some unrelated file operation, long after the code that leaked. The failing path is innocent; it is merely where the table ran out. Production limits are far higher than 64 (this sandbox’s was 20,000, see 01-environment.txt), which only postpones the failure from seconds to hours. On Linux, counting /proc/<pid>/fd over time, as the demo does, shows the slope before the crash.
One safety trap when walking: Files.walk does not follow symbolic links by default, so a link that points back up the tree is harmless. If you pass FileVisitOption.FOLLOW_LINKS, a loop is detected and reported as FileSystemLoopException, as the last two lines of the walk transcript show:
walk with a symlink loop, default: 11 entries, no error
walk with FOLLOW_LINKS: FileSystemLoopException
Files.walk, Files.find, Files.walkFileTree in the Files Javadoc
WatchService tells you that something changed, not exactly what happened
Polling a directory every few seconds to see whether a file appeared is wasteful, and WatchService is NIO.2’s answer: you register a directory, then block on take() or poll() until the operating system reports a change. On Linux the JDK builds it on inotify, the kernel’s file-notification facility. The shape of the API is: register a Path for the kinds of events you care about, get a WatchKey, call pollEvents() to collect what happened, and call reset() on the key or you will never hear from that directory again.
The diagram is a summary of what WatchDemo.java observed, scenario by scenario. First the simple cases (10-watch.txt):
System.out.println("1. create a file, write it twice, delete it (events polled afterwards):");
Path f = dir.resolve("a.txt");
Files.writeString(f, "one");
Files.writeString(f, "two", StandardOpenOption.APPEND);
Files.delete(f);
drain(ws, 300, true);
System.out.println("2. Files.writeString of 100 KiB to a new file (one call):");
Files.writeString(dir.resolve("b.txt"), "y".repeat(100 * 1024));
drain(ws, 300, true);
System.out.println("3. 50 appends to one file with no polling in between, then poll:");
Path g = dir.resolve("g.txt");
Files.writeString(g, "");
drain(ws, 300, false);
for (int i = 0; i < 50; i++) Files.writeString(g, "line\n", StandardOpenOption.APPEND);
System.out.println(" " + drain(ws, 300, false));
System.out.println("4. Files.move (atomic rename) of g.txt to h.txt:");
Files.move(g, dir.resolve("h.txt"), StandardCopyOption.ATOMIC_MOVE);
drain(ws, 300, true);
System.out.println("5. a file created inside a NEW subdirectory (subdirectory not registered):");
Path sub = Files.createDirectory(dir.resolve("sub"));
Files.writeString(sub.resolve("deep.txt"), "hello");
drain(ws, 300, true);
1. create a file, write it twice, delete it (events polled afterwards):
ENTRY_CREATE count=1 context=a.txt
ENTRY_MODIFY count=2 context=a.txt
ENTRY_DELETE count=1 context=a.txt
2. Files.writeString of 100 KiB to a new file (one call):
ENTRY_CREATE count=1 context=b.txt
ENTRY_MODIFY count=1 context=b.txt
3. 50 appends to one file with no polling in between, then poll:
{(batches)=1, ENTRY_MODIFY=9}
4. Files.move (atomic rename) of g.txt to h.txt:
ENTRY_DELETE count=1 context=g.txt
ENTRY_CREATE count=1 context=h.txt
5. a file created inside a NEW subdirectory (subdirectory not registered):
ENTRY_CREATE count=1 context=sub
Create, write, append, delete gave one ENTRY_CREATE, one ENTRY_MODIFY with count=2, and one ENTRY_DELETE. Repeated modifications of the same file arrive folded into a single event carrying a repeat count. Code that uses events.size() to count changes will be wrong; read event.count().
One Files.writeString of 100 KiB to a new file produced one ENTRY_CREATE and one ENTRY_MODIFY (count 1 in this run). How many modify events one logical write produces is not something to rely on; scenario 3 below shows the counts moving from run to run.
A rename is a delete plus a create. Moving g.txt to h.txt with ATOMIC_MOVE arrived as ENTRY_DELETE g.txt and ENTRY_CREATE h.txt, with no link between them.
Only the registered directory is watched. Creating sub was reported; creating sub/deep.txt afterwards was not. There is no built-in recursion, so a recursive watcher must register every new directory itself, typically from a walkFileTree pass plus handling of each ENTRY_CREATE for a directory.
Scenario 3 deserves its own look, because the output is different on every run. Fifty appends to one file, with no polling in between, came out as a total ENTRY_MODIFY count of 9 in the main transcript, and five further runs in 11-watch-tuned-and-repeat.txt gave:
Scenario 3 (50 appends, no polling) over five more runs:
{(batches)=1, ENTRY_MODIFY=21}
{(batches)=1, ENTRY_MODIFY=20}
{(batches)=1, ENTRY_MODIFY=2}
{(batches)=1, ENTRY_MODIFY=1}
{(batches)=1, ENTRY_MODIFY=19}
No run reported anywhere near 50. The JDK code (excerpt in the accordion below) can only add to a count, never subtract, so the missing modifications were merged before they reached it. I suspect the kernel, which is known to merge identical unread inotify events, but I did not verify that, so take it as a suspicion. The consequence is firm either way: WatchService is a prompt to look, not a log of every write.
The overflow cliff: the 513th pending event destroys the other 512
Every WatchKey holds a list of pending events, and the JDK caps that list. Scenario 6 creates N empty files in a watched directory without polling until the end:
From WatchDemo.java (output: 10-watch.txt):
for (int n : new int[] {100, 512, 513, 5000}) {
Path burst = Files.createDirectory(dir.resolve("burst" + n));
try (WatchService ws2 = FileSystems.getDefault().newWatchService()) {
burst.register(ws2, ENTRY_CREATE);
for (int i = 0; i < n; i++) Files.createFile(burst.resolve("f" + i));
Thread.sleep(500);
WatchKey k = ws2.poll(1, TimeUnit.SECONDS);
List<WatchEvent<?>> evs = k.pollEvents();
long creates = evs.stream().filter(e -> e.kind() == ENTRY_CREATE).count();
long overflow = evs.stream().filter(e -> e.kind() == OVERFLOW).mapToInt(WatchEvent::count).sum();
System.out.printf(" N=%-5d delivered %-4d events: ENTRY_CREATE=%-4d OVERFLOW(count)=%d, files on disk=%d%n",
n, evs.size(), creates, overflow, Files.list(burst).count());
}
}
6. burst: create N empty files in a fresh directory before polling once (jdk.nio.file.WatchService.maxEventsPerPoll=default):
N=100 delivered 100 events: ENTRY_CREATE=100 OVERFLOW(count)=0, files on disk=100
N=512 delivered 512 events: ENTRY_CREATE=512 OVERFLOW(count)=0, files on disk=512
N=513 delivered 1 events: ENTRY_CREATE=0 OVERFLOW(count)=1, files on disk=513
N=5000 delivered 1 events: ENTRY_CREATE=0 OVERFLOW(count)=4488, files on disk=5000
At 512 files every event is delivered. At 513, the result is one OVERFLOW event and no ENTRY_CREATE at all: the 512 events already queued were discarded. At 5,000 files the single OVERFLOW carries count=4488, which is 5,000 minus the 512 that were discarded, and all 5,000 files exist on disk. The kernel’s own queue was not the limit here (max_queued_events was 16,384 in 01-environment.txt); the cap is in the JDK’s per-key list, and the JDK source shows why.
Reference: the JDK source that produces this behaviour (AbstractWatchKey.signalEvent)
This is a verbatim extract from the src.zip of the JDK used for every run (12-watchkey-source-excerpt.txt). Read the three branches in order: repeat of the same event or context becomes increment(); a modify for a context that already has a pending modify becomes increment(); and once the list reaches MAX_EVENT_LIST_SIZE, the new event is turned into OVERFLOW and events.clear() drops everything pending.
final void signalEvent(WatchEvent.Kind<?> kind, Object context) {
boolean isModify = (kind == StandardWatchEventKinds.ENTRY_MODIFY);
synchronized (this) {
int size = events.size();
if (size > 0) {
// if the previous event is an OVERFLOW event or this is a
// repeated event then we simply increment the counter
WatchEvent<?> prev = events.get(size-1);
if ((prev.kind() == StandardWatchEventKinds.OVERFLOW) ||
((kind == prev.kind() &&
Objects.equals(context, prev.context()))))
{
((Event<?>)prev).increment();
return;
}
// if this is a modify event and the last entry for the context
// is a modify event then we simply increment the count
if (!lastModifyEvents.isEmpty()) {
if (isModify) {
WatchEvent<?> ev = lastModifyEvents.get(context);
if (ev != null) {
assert ev.kind() == StandardWatchEventKinds.ENTRY_MODIFY;
((Event<?>)ev).increment();
return;
}
} else {
// not a modify event so remove from the map as the
// last event will no longer be a modify event.
lastModifyEvents.remove(context);
}
}
// if the list has reached the limit then drop pending events
// and queue an OVERFLOW event
if (size >= MAX_EVENT_LIST_SIZE) {
kind = StandardWatchEventKinds.OVERFLOW;
isModify = false;
context = null;
}
}
// non-repeated event
Event<Object> ev =
new Event<>((WatchEvent.Kind<Object>)kind, context);
if (isModify) {
lastModifyEvents.put(context, ev);
} else if (kind == StandardWatchEventKinds.OVERFLOW) {
// drop all pending events
events.clear();
lastModifyEvents.clear();
}
events.add(ev);
signal();
The limit is configurable through the system property jdk.nio.file.WatchService.maxEventsPerPoll, which the same source reads (default 512). With it set to 10000, the same burst test delivered all N events (11-watch-tuned-and-repeat.txt):
$ java -Djdk.nio.file.WatchService.maxEventsPerPoll=10000 WatchDemo (scenario 6 only)
6. burst: create N empty files in a fresh directory before polling once (jdk.nio.file.WatchService.maxEventsPerPoll=10000):
N=100 delivered 100 events: ENTRY_CREATE=100 OVERFLOW(count)=0, files on disk=100
N=512 delivered 512 events: ENTRY_CREATE=512 OVERFLOW(count)=0, files on disk=512
N=513 delivered 513 events: ENTRY_CREATE=513 OVERFLOW(count)=0, files on disk=513
N=5000 delivered 5000 events: ENTRY_CREATE=5000 OVERFLOW(count)=0, files on disk=5000
Raising the limit moves the cliff, it does not remove it, and it was a property I only found by reading source, so check it still exists on your JDK version before depending on it.
What to do with this. The Javadoc tells you the same thing in general terms: an implementation may “impose an unspecified limit on the number of events that it may accumulate” and, when events are discarded, pollEvents() returns an OVERFLOW event “which can be used by the consumer as a trigger to re-examine the state of the object”. So treat every OVERFLOW as “rescan the directory”, drain the key promptly (do the slow work on another thread), and design the consumer so that a missed event is repaired by the next scan. Use WatchService for configuration reload or a drop-folder where you can rescan; do not use it as an audit trail. Other operating systems may behave differently, which I did not test.
WatchService Javadoc (the paragraphs on overflow and on modification-in-progress are worth reading in full)
Memory-mapped files keep the heap flat, but the memory still has to live somewhere
The last technique maps the file into the process’s address space: the operating system pages bytes in from disk when you touch them and you read them like an array, with no read() calls and no copy onto the Java heap. FileChannel.map is the classic way to do this and returns a MappedByteBuffer. It is most useful for random access to a big file, such as an index you jump around in, rather than a front-to-back scan.
The classic API has a hard wall: a ByteBuffer is indexed by an int, so one mapping cannot exceed 2 GiB. Try to map the whole 5 GB file and this is what you get; the demo then maps it in 1 GiB windows, and separately maps it in one piece with the newer Arena/MemorySegment API (the foreign function and memory API, final in Java 22 through JEP 454) that uses long offsets:
From MappedDemo.java (output: 13-mapped.txt):
/** Counts '\n' with an Arena-backed mapping of the whole file: no 2 GiB limit, no heap copy. */
public static long countNewlinesArena(Path p) throws IOException {
try (Arena arena = Arena.ofConfined();
FileChannel ch = FileChannel.open(p, StandardOpenOption.READ)) {
MemorySegment seg = ch.map(FileChannel.MapMode.READ_ONLY, 0, ch.size(), arena);
long n = 0;
for (long i = 0, len = seg.byteSize(); i < len; i++) if (seg.get(ValueLayout.JAVA_BYTE, i) == '\n') n++;
return n;
}
}
/** Same with the classic API, in windows of at most 1 GiB. */
public static long countNewlinesWindows(Path p) throws IOException {
long n = 0;
try (FileChannel ch = FileChannel.open(p, StandardOpenOption.READ)) {
long size = ch.size(), win = 1L << 30;
for (long off = 0; off < size; off += win) {
MappedByteBuffer mb = ch.map(FileChannel.MapMode.READ_ONLY, off, Math.min(win, size - off));
for (int i = 0, lim = mb.limit(); i < lim; i++) if (mb.get(i) == '\n') n++;
}
}
return n;
}
$ java -Xmx64m MappedDemo big.log
file big.log: 5,000,000,069 bytes, -Xmx=64 MiB, RSS at start=40 MiB
classic map of the whole file: IllegalArgumentException: Size exceeds Integer.MAX_VALUE
Arena + MemorySegment: 34,120,522 newlines in 5.1 s; RSS after the Arena was closed=51 MiB
classic 1 GiB windows: 34,120,522 newlines in 7.8 s; RSS right after=4821 MiB
heap: -Xmx=64 MiB, peak used (sampled every 20 ms)=4 MiB, GC collections=0, GC time=0 ms
RSS one second after System.gc()=52 MiB
The results, at -Xmx64m: the whole-file classic map fails with Size exceeds Integer.MAX_VALUE; both ways of counting newlines agree with each other and with the 34,120,522 lines the streaming code counted; and the heap peak was 4 MiB with zero garbage collections, because nothing was copied onto the heap. That is the real advantage. A test in the repo checks the 2 GiB wall without a 3 GB file, by mapping a sparse file (Nio2BehaviourTest).
Now read the resident-memory numbers, because they are the catch. “RSS” is how much physical memory the process currently has mapped in. After the Arena version finished and its arena was closed, RSS was back to about 51 MiB. After the classic windows version it was 4,821 MiB, about 4.7 GiB (the RSS right after figure), and it only dropped once System.gc() ran. The Javadoc explains why: a mapped byte buffer and its mapping “remain valid until the buffer itself is garbage-collected”. Here there were no collections, because the heap was hardly used, so the mappings stayed. The Arena is closed deterministically by try-with-resources.
The trap. A process that maps a lot through MappedByteBuffer and allocates little heap can hold gigabytes of mappings that the garbage collector has no reason to release. The pages are file-backed and the operating system can evict them under pressure, so this is not an out-of-memory condition by itself, but it does show up in RSS, in container memory accounting, and in anything that watches it. If you map big files, prefer the Arena API so the lifetime is explicit.
Honest advice. Most code should use Files.lines or a BufferedReader and nothing more exotic: they are the option that failed in none of the experiments above. Memory mapping is an optimisation for a specific access pattern and brings its own lifetime rules; do not reach for it to make a sequential read faster without measuring on your own hardware, because I measured only newline counting on one shared VM, where the mapped version was faster than the stream (5.1 s against 9.4 s in the transcripts) but did a different and simpler job. A WatchService is not a reliable event log. And everything here about WatchService and descriptor counts is Linux only; I did not run it on Windows or macOS, or on a network file system, where the behaviour may differ.
Reference: the 12 assertions that pin these claims
Each behaviour above that does not need a 5 GB file also has a test that fails if the JDK changes it, all in Nio2BehaviourTest.java: streaming agrees with readAllLines; the 2 GiB classic-map wall and the Arena success on a sparse 3 GiB file; descriptor counts for unclosed Files.lines and early-exit Files.walk; lazy UncheckedIOException on a bad byte; walk/find; WatchService create event and modify folding; the 512-pending overflow; no recursion; and a child-JVM test that readAllLines fails at -Xmx32m where streaming succeeds. The result file is 14-tests.txt:
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.755 s -- in com.ankurm.nio2.Nio2BehaviourTest
No Comments yet!