asmhatre/java-core-examples
1
0
Fork 0
Code Activity

bufferedreader: read a file line by line module (BufferedReader vs Scanner vs Files.lines vs readAllLines on a 1 GB file; heap at -Xmx64m, timings, read-call counts, one-huge-line OOM, charset/BOM/line-terminator pitfalls, Scanner swallowing IOException), 10 contract tests

Browse Source 
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_017ic4b3pDoGLiWk6LLBeHjW
This commit is contained in:
Claude
2026-11-15 01:20:37 +05:30
parent 5faa57bb2d
commit f8c9e244a3
25 changed files with 809 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
bufferedreader/output/01-environment.txt
+11
View File
@@ -0,0 +1,11 @@
$ uname -sr
Linux 6.18.44-fc-v470
$ nproc
2
$ java -version
openjdk version "25.0.5" 2026-10-20 LTS
OpenJDK Runtime Environment Temurin-25.0.5+8 (build 25.0.5+8-LTS)
OpenJDK 64-Bit Server VM Temurin-25.0.5+8 (build 25.0.5+8-LTS, mixed mode, sharing)
$ free -m (before)
total used free shared buff/cache available
Mem: 8024 615 6866 16 784 7409
bufferedreader/output/02-generate.txt
+13
View File
@@ -0,0 +1,13 @@
generated big.log: 1,000,000,064 bytes in 9620 ms
generated medium.log: 200,000,066 bytes in 2297 ms
generated long.txt: 200,000,071 bytes in 237 ms
$ ls -l (generated files)
1000000064 big.log
200000071 long.txt
200000066 medium.log
$ head -3 big.log
2026-01-01T00:00:00.000Z INFO [worker-00] request id=000000000 path=/api/v1/items/0 status=500 ms=0
2026-01-01T00:00:01.001Z INFO [worker-01] request id=000000001 path=/api/v1/items/1 status=200 ms=1
2026-01-01T00:00:02.002Z INFO [worker-02] request id=000000002 path=/api/v1/items/2 status=200 ms=2
$ wc -l big.log
9623347
bufferedreader/output/03-heap.txt
+22
View File
@@ -0,0 +1,22 @@
$ java -Xmx64m LineCount reader big.log
reader lines=9,623,347 chars=990,376,717 time=1,288 ms gc=37 collections / 31 ms (-Xmx64 MB)
$ java -Xmx64m LineCount scanner big.log
scanner lines=9,623,347 chars=990,376,717 time=6,233 ms gc=140 collections / 116 ms (-Xmx64 MB)
$ java -Xmx64m LineCount lines big.log
lines lines=9,623,347 chars=990,376,717 time=1,179 ms gc=37 collections / 32 ms (-Xmx64 MB)
$ java -Xmx64m LineCount readAllLines big.log
readAllLines OutOfMemoryError: Java heap space (-Xmx64 MB)
How much heap does readAllLines need for this file?
$ java -Xmx1g LineCount readAllLines big.log
readAllLines OutOfMemoryError: Java heap space (-Xmx1024 MB)
$ java -Xmx2g LineCount readAllLines big.log
readAllLines lines=9,623,347 chars=990,376,717 time=2,872 ms gc=24 collections / 922 ms (-Xmx2048 MB)
$ java -Xmx3g LineCount readAllLines big.log
readAllLines lines=9,623,347 chars=990,376,717 time=2,729 ms gc=23 collections / 949 ms (-Xmx3072 MB)
$ java -Xmx4g LineCount readAllLines big.log
readAllLines lines=9,623,347 chars=990,376,717 time=2,862 ms gc=23 collections / 905 ms (-Xmx4096 MB)
And the streaming reader at a tiny heap:
$ java -Xmx16m LineCount reader big.log
reader lines=9,623,347 chars=990,376,717 time=1,347 ms gc=171 collections / 101 ms (-Xmx16 MB)
bufferedreader/output/04-timing.txt
+19
View File
@@ -0,0 +1,19 @@
$ java -Xmx5g LineCount <approach> big.log (round-robin, 3 rounds, fresh JVM each)
-- round 1
reader lines=9,623,347 chars=990,376,717 time=1,334 ms gc=18 collections / 21 ms (-Xmx5120 MB)
reader64k lines=9,623,347 chars=990,376,717 time=1,260 ms gc=18 collections / 21 ms (-Xmx5120 MB)
scanner lines=9,623,347 chars=990,376,717 time=5,367 ms gc=49 collections / 49 ms (-Xmx5120 MB)
lines lines=9,623,347 chars=990,376,717 time=1,311 ms gc=18 collections / 24 ms (-Xmx5120 MB)
readAllLines lines=9,623,347 chars=990,376,717 time=2,892 ms gc=23 collections / 1048 ms (-Xmx5120 MB)
-- round 2
reader lines=9,623,347 chars=990,376,717 time=1,216 ms gc=19 collections / 20 ms (-Xmx5120 MB)
reader64k lines=9,623,347 chars=990,376,717 time=1,233 ms gc=18 collections / 21 ms (-Xmx5120 MB)
scanner lines=9,623,347 chars=990,376,717 time=5,645 ms gc=50 collections / 51 ms (-Xmx5120 MB)
lines lines=9,623,347 chars=990,376,717 time=1,341 ms gc=17 collections / 26 ms (-Xmx5120 MB)
readAllLines lines=9,623,347 chars=990,376,717 time=2,788 ms gc=23 collections / 955 ms (-Xmx5120 MB)
-- round 3
reader lines=9,623,347 chars=990,376,717 time=1,254 ms gc=18 collections / 21 ms (-Xmx5120 MB)
reader64k lines=9,623,347 chars=990,376,717 time=1,325 ms gc=18 collections / 21 ms (-Xmx5120 MB)
scanner lines=9,623,347 chars=990,376,717 time=5,300 ms gc=49 collections / 45 ms (-Xmx5120 MB)
lines lines=9,623,347 chars=990,376,717 time=1,398 ms gc=18 collections / 25 ms (-Xmx5120 MB)
readAllLines lines=9,623,347 chars=990,376,717 time=2,985 ms gc=24 collections / 1236 ms (-Xmx5120 MB)
bufferedreader/output/05-read-calls.txt
+17
View File
@@ -0,0 +1,17 @@
== 1. read() calls that reach the underlying InputStream (first 50 MB of the file) ==
InputStreamReader.read() per char underlying calls= 6,104 avg bytes/call= 8,192.0
BufferedReader(8192).read() per char underlying calls= 6,104 avg bytes/call= 8,192.0
BufferedReader(8192).readLine() underlying calls= 6,104 avg bytes/call= 8,192.0
BufferedReader(65536).readLine() underlying calls= 6,104 avg bytes/call= 8,192.0
BufferedReader(1).readLine() underlying calls= 6,104 avg bytes/call= 8,192.0
== 2. time to consume the whole file (3 passes each; indicative, same JVM) ==
pass 1 InputStreamReader.read() per char 6,493 ms
pass 1 BufferedReader.read() per char 4,902 ms
pass 1 BufferedReader.readLine() 206 ms
pass 2 InputStreamReader.read() per char 7,373 ms
pass 2 BufferedReader.read() per char 4,948 ms
pass 2 BufferedReader.readLine() 279 ms
pass 3 InputStreamReader.read() per char 7,395 ms
pass 3 BufferedReader.read() per char 5,077 ms
pass 3 BufferedReader.readLine() 269 ms
bufferedreader/output/06-long-line.txt
+6
View File
@@ -0,0 +1,6 @@
$ java -Xmx128m LongLineDemo line long.txt
mode=line -Xmx128 MB
OutOfMemoryError: Java heap space
$ java -Xmx128m LongLineDemo chunks long.txt
mode=chunks -Xmx128 MB
read(char[]) processed 200,000,071 chars, 2,739,727 commas, never holding more than 8192 chars
bufferedreader/output/07-charset.txt
+22
View File
@@ -0,0 +1,22 @@
$ java CharsetDemo
Charset.defaultCharset() = UTF-8
native.encoding = ANSI_X3.4-1968
file.encoding (property) = UTF-8
latin1.txt bytes: 63 61 66 e9 0a (the text is "café" saved as ISO-8859-1)
new FileReader(path) -> "caf\uFFFD"
new FileReader(path, UTF_8) -> "caf\uFFFD"
Files.newBufferedReader(path) -> MalformedInputException: Input length = 1
Files.readAllLines(path) -> MalformedInputException: Input length = 1
Files.lines(path).toList() -> UncheckedIOException: java.nio.charset.MalformedInputException: Input length = 1 [cause: MalformedInputException: Input length = 1]
new Scanner(path).nextLine() -> "caf\uFFFD"
readAllLines(path, ISO_8859_1) <- fix -> [café]
utf8.txt bytes: 63 61 66 c3 a9 20 e2 82 b9 31 30 30 0a ("café ₹100" saved as UTF-8)
new FileReader(path) -> "caf\u00E9 \u20B9100"
new FileReader(path, UTF_8) <- fix -> "caf\u00E9 \u20B9100"
bom.csv bytes: ef bb bf 69 64 2c 6e 61 6d 65 0a 31 2c 41 ...
header line = "\uFEFFid,name" length=8 (the text is 7 characters)
header.equals("id,name") = false
after stripping U+FEFF = "id,name" equals=true
bufferedreader/output/08-charset-compat.txt
+22
View File
@@ -0,0 +1,22 @@
$ LC_ALL=C java -Dfile.encoding=COMPAT CharsetDemo
Charset.defaultCharset() = US-ASCII
native.encoding = ANSI_X3.4-1968
file.encoding (property) = ANSI_X3.4-1968
latin1.txt bytes: 63 61 66 e9 0a (the text is "café" saved as ISO-8859-1)
new FileReader(path) -> "caf\uFFFD"
new FileReader(path, UTF_8) -> "caf\uFFFD"
Files.newBufferedReader(path) -> MalformedInputException: Input length = 1
Files.readAllLines(path) -> MalformedInputException: Input length = 1
Files.lines(path).toList() -> UncheckedIOException: java.nio.charset.MalformedInputException: Input length = 1 [cause: MalformedInputException: Input length = 1]
new Scanner(path).nextLine() -> "caf\uFFFD"
readAllLines(path, ISO_8859_1) <- fix -> [café]
utf8.txt bytes: 63 61 66 c3 a9 20 e2 82 b9 31 30 30 0a ("café ₹100" saved as UTF-8)
new FileReader(path) -> "caf\uFFFD\uFFFD \uFFFD\uFFFD\uFFFD100"
new FileReader(path, UTF_8) <- fix -> "caf\u00E9 \u20B9100"
bom.csv bytes: ef bb bf 69 64 2c 6e 61 6d 65 0a 31 2c 41 ...
header line = "\uFEFFid,name" length=8 (the text is 7 characters)
header.equals("id,name") = false
after stripping U+FEFF = "id,name" equals=true
bufferedreader/output/09-line-endings.txt
+10
View File
@@ -0,0 +1,10 @@
input | BufferedReader.readLine() | Scanner.nextLine() | split("\n").length
--------------------------------------------------------------------------------------------------------
"a\nb" | ["a", "b"] | ["a", "b"] | 2
"a\nb\n" | ["a", "b"] | ["a", "b"] | 2
"a\nb\n\n" | ["a", "b", ""] | ["a", "b", ""] | 2
"a\r\nb" | ["a", "b"] | ["a", "b"] | 2
"a\rb" | ["a", "b"] | ["a", "b"] | 1
"" | [] | [] | 1
"\n" | [""] | [""] | 0
"a\n\rb" | ["a", "", "b"] | ["a", "", "b"] | 2
bufferedreader/output/10-scanner-swallows.txt
+9
View File
@@ -0,0 +1,9 @@
-- BufferedReader --
got: line one
got: line two
IOException after 2 lines: simulated read failure (disk gone)
-- Scanner --
got: line one
got: line two
loop ended normally after 2 lines
sc.ioException() = java.io.IOException: simulated read failure (disk gone)
bufferedreader/output/11-tests.txt
+4
View File
@@ -0,0 +1,4 @@
-------------------------------------------------------------------------------
Test set: com.ankurm.bufferedreader.ReadLineContractTest
-------------------------------------------------------------------------------
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.194 s -- in com.ankurm.bufferedreader.ReadLineContractTest