Add file-transfer module: multipart limits, filename handling, Range downloads, streaming under -Xmx64m, S3 presigned URLs
Co-Authored-By: Claude Sonnet 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01VRLbHYdXVEBzGVhYq3pz4q
This commit is contained in:
@@ -42,6 +42,7 @@ files.
|
||||
| [`kotlin/`](kotlin) | [Kotlin with Spring Boot 4.1: Null Safety, Coroutines, and What Java Developers Get Wrong](https://ankurm.com/) | Kotlin 2.3.21 (Boot 4.1.1's own managed version, not the latest GA), the real `@Configuration class may not be final` and `@Bean method must not be private or final` failures without the allopen/noarg `spring`/`jpa` compiler plugins, `suspend fun`/`Flow<T>` controller methods proven to still resume off a Tomcat thread under plain `spring-boot-starter-webmvc`, a JSpecify `@NullMarked` Java method that still NPEs at runtime because the contract isn't runtime-checked, and `TestRestTemplate` confirmed absent from Boot 4.1's `spring-boot-test` jar |
|
||||
| [`testing-no-containers/`](testing-no-containers) | [Testing Spring Boot 4 Without Containers: WireMock, MockWebServer and @MockitoBean](https://ankurm.com/) | `@MockBean`'s total removal proven with a real `javac` failure, `@MockitoBean`/`@MockitoSpyBean` context-cache behavior proven with real `org.springframework.test.context.cache` DEBUG log lines, WireMock fault injection against a real `RestClient`, a real `NoClassDefFoundError` from mixing `wiremock` with `wiremock-jetty12` (fixed with `wiremock-standalone`), and MockWebServer's `SocketEffect.ShutdownConnection` for genuinely severing a connection mid-response |
|
||||
| [`rest-test-client/`](rest-test-client) | [RestTestClient in Spring Framework 7: Testing REST APIs End-to-End (vs MockMvcTester)](https://ankurm.com/resttestclient-spring-framework-7-end-to-end-testing/) | one fluent `RestTestClient` API proven across `bindToController`, `bindTo(mockMvc)`, `bindToApplicationContext` and `bindToServer`; `RestTestClient`, `TestRestTemplate` and `@WebMvcTest` confirmed relocated into new Spring Boot 4 modules by unzipping the real jars; two distinct real failures for the two now-separate `@AutoConfigureRestTestClient`/`@AutoConfigureTestRestTemplate` annotations; and the JDK `HttpClient` Boot picks for `bindToServer` caught transparently decoding gzip and hiding `Content-Encoding`, verified against `curl` and against the older `SimpleClientHttpRequestFactory` |
|
||||
| [`file-transfer/`](file-transfer) | [File Upload and Download in Spring Boot 4: Multipart, Streaming and S3 Presigned URLs](https://ankurm.com/file-upload-download-spring-boot-4-multipart-streaming-s3-presigned-urls/) | multipart limits enforced by Tomcat (and ignored by `MockMvc`), the client-controlled filename, `Range` downloads, a 200 MB download and 150 MB upload under `-Xmx64m`, and S3 presigned URLs checked against a local SigV4 verifier (not a real S3) |
|
||||
|
||||
`core-di`, `core-beans`, `custom-starter`, `core-events`, `multi-datasource`, `i18n`, `aot-cache`, `kotlin`, `testing-no-containers`, `rest-test-client` and `hexagonal` are the exception: they have **no `docs/NN-topic.md` chapter folder**. Their deeper material lives in
|
||||
collapsible sections inside the articles themselves. Where their captured output sits is not
|
||||
|
||||
@@ -0,0 +1,2 @@
|
||||
target/
|
||||
cap/app.log
|
||||
@@ -0,0 +1,25 @@
|
||||
# file-transfer
|
||||
|
||||
Companion code for **[File Upload and Download in Spring Boot 4: Multipart, Streaming and S3 Presigned URLs](https://ankurm.com/file-upload-download-spring-boot-4-multipart-streaming-s3-presigned-urls/)**.
|
||||
|
||||
Spring Boot 4.1.1, Spring Framework 7.0.9, Tomcat 11.0.24, JDK 25, AWS SDK for Java 2.56.0. No Docker needed.
|
||||
|
||||
| Area | Code | What the run shows |
|
||||
|---|---|---|
|
||||
| Multipart limits | `UploadController`, `UploadErrors`, `MultipartLimitsTest` | the 1 MB / 2 MB limits are enforced by Tomcat, so `MockMvc` accepts a file the real server rejects; the rejected-upload race (`cap/01-limits.txt`) |
|
||||
| Filenames | `FileStore`, `FilenameAndDownloadTest` | `getOriginalFilename()` is whatever the client sent, including `../../`; the file is stored under a UUID (`cap/02-filename.txt`) |
|
||||
| Download | `DownloadController` | `Resource` gives `Range`/206/416 for free; `Content-Disposition` encoding (`cap/03-download.txt`) |
|
||||
| Presigned URLs | `S3PresignService`, `PresignController`, `FakeS3`, `PresignedUrlTest` | what is and is not covered by the signature; expiry; reuse (`cap/04-presign.txt`) |
|
||||
| Streaming | `StreamController`, `scripts/run-all.sh` | 200 MB download and a 150 MB upload under `-Xmx64m` (`cap/05-streaming.txt`, `cap/06-big-upload.txt`) |
|
||||
|
||||
```bash
|
||||
./scripts/run-all.sh # tests, then the -Xmx64m experiments; rewrites cap/*.txt
|
||||
```
|
||||
|
||||
## What was and was not run
|
||||
|
||||
- The presigned URLs are produced by the real AWS SDK presigner and checked by `FakeS3`, a small in-process server
|
||||
that recomputes the AWS Signature V4 query signature from the published algorithm. That proves the URLs are consistent
|
||||
with the algorithm as documented. It does **not** prove a real S3 or MinIO server accepts them: this environment has
|
||||
no Docker and no S3-compatible server, so nothing here was run against one.
|
||||
- `app.s3.access-key` and `app.s3.secret-key` are the example keys from the AWS documentation. They are not credentials.
|
||||
@@ -0,0 +1,6 @@
|
||||
limits: max-file-size=1MB, max-request-size=2MB
|
||||
real HTTP, one 500 KB file -> status=201 body={"id":"d8d0b4b8-d530-42aa-b005-9e1b747449a7","originalFilename":"a.txt","displayName":"a.txt","size":512000,"declaredContentType":"text/plai...
|
||||
real HTTP, one 1.5 MB file -> status=413 body={"detail":"The upload is larger than the configured limit (spring.servlet.multipart.max-file-size / max-request-size).","instance":"/files",...
|
||||
real HTTP, 3 x 900 KB in one request -> status=413 body={"detail":"The upload is larger than the configured limit (spring.servlet.multipart.max-file-size / max-request-size).","instance":"/files",...
|
||||
real HTTP, one 10 MB file, 10 attempts -> 7 x status=413, 3 x connection error
|
||||
MockMvc, the same 1.5 MB file -> status=201
|
||||
@@ -0,0 +1,4 @@
|
||||
client sent filename=../../etc/cron.d/evil.txt Content-Type=image/png
|
||||
server response: {"id":"4980219c-b92a-46d4-a5ab-b5d7636fb7f0","originalFilename":"../../etc/cron.d/evil.txt","displayName":"evil.txt","size":5,"declaredContentType":"image/png"}
|
||||
root.resolve(originalFilename).normalize() stays inside the storage dir: false
|
||||
client sent a Windows-style path; server response: {"id":"d436e57e-6855-4fff-b5ca-b9dd0fec25c2","originalFilename":"..\\..\\windows\\system32\\evil.dll","displayName":"evil.dll","size":1,"declaredContentType":"text/plain"}
|
||||
@@ -0,0 +1,12 @@
|
||||
GET /files/{id}
|
||||
status=200 body=ABCDEFGHIJKLMNOPQRSTUVWXYZ
|
||||
content-type: application/octet-stream
|
||||
content-length: 26
|
||||
content-disposition: attachment; filename="resume 2026.txt"; filename*=UTF-8''r%C3%A9sum%C3%A9%202026.txt
|
||||
accept-ranges: bytes
|
||||
x-content-type-options: nosniff
|
||||
GET /files/{id} with Range: bytes=10-19
|
||||
status=206 body=KLMNOPQRST
|
||||
content-range: bytes 10-19/26
|
||||
GET /files/{id} with Range: bytes=500-
|
||||
status=416 content-range: bytes */26
|
||||
@@ -0,0 +1,18 @@
|
||||
PUT a URL signed for 1 second, sent after 2.2 seconds -> status=403 <Error><Code>AccessDenied</Code></Error>
|
||||
POST /presign/upload {"contentType":"image/png"} -> status=200
|
||||
method=PUT expiresInSeconds=300 headers={"content-type":"image/png"}
|
||||
url path: http://<fake-s3>/uploads/4dfdf0af-141a-4d1b-9d35-3609a9385791
|
||||
query: X-Amz-Algorithm=AWS4-HMAC-SHA256
|
||||
query: X-Amz-Date=20261017T154013Z
|
||||
query: X-Amz-SignedHeaders=content-type%3Bhost
|
||||
query: X-Amz-Credential=AKIAIOSFODNN7EXAMPLE%2F20261017%2Fus-east-1%2Fs3%2Faws4_request
|
||||
query: X-Amz-Expires=300
|
||||
query: X-Amz-Signature=4e5c56c29980cf521e477119788a0616c392deac4686df7259957af9f1445203
|
||||
PUT with the signed Content-Type (image/png) -> status=200
|
||||
PUT the same URL again (presigned URLs are reusable) -> status=200
|
||||
PUT with a different Content-Type (text/html) -> status=403 <Error><Code>SignatureDoesNotMatch</Code></Error>
|
||||
PUT with no Content-Type header -> status=403 <Error><Code>SignatureDoesNotMatch</Code></Error>
|
||||
PUT after changing the last character of the key -> status=403 <Error><Code>SignatureDoesNotMatch</Code></Error>
|
||||
PUT after raising X-Amz-Expires to 7 days -> status=403 <Error><Code>SignatureDoesNotMatch</Code></Error>
|
||||
GET presigned URL -> status=200 bytes match upload=true content-disposition=attachment; filename="holiday.png"
|
||||
disallowed type: POST /presign/upload text/html -> status=400
|
||||
@@ -0,0 +1,4 @@
|
||||
JVM started with -Xmx64m
|
||||
GET /buffered/200 (200 MB) -> status=500 downloaded=108 bytes in 0.273337s
|
||||
GET /stream/200 (200 MB) -> status=200 downloaded=209715200 bytes in 0.337059s
|
||||
GET /stream/1 afterwards -> status=200
|
||||
@@ -0,0 +1,3 @@
|
||||
JVM started with -Xmx64m, multipart limits removed, uploading a 150 MB file
|
||||
multipart POST /files -> status=201 sent=157286616 bytes in 1.518657s
|
||||
raw POST /files/raw -> status=201 sent=157286400 bytes in 1.850810s
|
||||
@@ -0,0 +1,67 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<project xmlns="http://maven.apache.org/POM/4.0.0"
|
||||
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
||||
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||
<modelVersion>4.0.0</modelVersion>
|
||||
|
||||
<parent>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter-parent</artifactId>
|
||||
<version>4.1.1</version>
|
||||
<relativePath/>
|
||||
</parent>
|
||||
|
||||
<groupId>com.ankurm</groupId>
|
||||
<artifactId>file-transfer</artifactId>
|
||||
<version>1.0.0</version>
|
||||
<name>file-transfer</name>
|
||||
<description>File upload and download in Spring Boot 4.1: multipart limits, filename handling, Range
|
||||
downloads, StreamingResponseBody under a 64 MB heap, and S3 presigned URLs checked against a local
|
||||
SigV4 verifier.</description>
|
||||
<packaging>jar</packaging>
|
||||
|
||||
<properties>
|
||||
<java.version>25</java.version>
|
||||
<awssdk.version>2.56.0</awssdk.version>
|
||||
</properties>
|
||||
|
||||
<dependencyManagement>
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>software.amazon.awssdk</groupId>
|
||||
<artifactId>bom</artifactId>
|
||||
<version>${awssdk.version}</version>
|
||||
<type>pom</type>
|
||||
<scope>import</scope>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
</dependencyManagement>
|
||||
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter-webmvc</artifactId>
|
||||
</dependency>
|
||||
|
||||
<!-- Only the presigner is used: it signs locally and never opens a connection. -->
|
||||
<dependency>
|
||||
<groupId>software.amazon.awssdk</groupId>
|
||||
<artifactId>s3</artifactId>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter-webmvc-test</artifactId>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
|
||||
<build>
|
||||
<plugins>
|
||||
<plugin>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-maven-plugin</artifactId>
|
||||
</plugin>
|
||||
</plugins>
|
||||
</build>
|
||||
</project>
|
||||
Executable
+36
@@ -0,0 +1,36 @@
|
||||
#!/usr/bin/env bash
|
||||
# Runs the whole module: the tests (which write cap/01..04), then the -Xmx64m streaming experiment (cap/05..06).
|
||||
# No Docker needed. Takes a couple of minutes.
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")/.."
|
||||
mkdir -p cap
|
||||
mvn -q -B test
|
||||
mvn -q -B package -DskipTests
|
||||
JAR=$(ls target/file-transfer-*.jar | grep -v original | head -1)
|
||||
|
||||
run_app() { # $1 = extra args; sets APP_PID
|
||||
java -Xmx64m -jar "$JAR" --server.port=18080 $1 > cap/app.log 2>&1 &
|
||||
APP_PID=$!
|
||||
for _ in $(seq 1 60); do curl -s -o /dev/null http://localhost:18080/stream/1 && break; sleep 0.5; done
|
||||
}
|
||||
|
||||
OUT=cap/05-streaming.txt; : > "$OUT"
|
||||
run_app ""
|
||||
echo "JVM started with -Xmx64m" >> "$OUT"
|
||||
for kind in buffered stream; do
|
||||
R=$(curl -s -o /dev/null -w "status=%{http_code} downloaded=%{size_download} bytes in %{time_total}s" http://localhost:18080/$kind/200 || true)
|
||||
echo "GET /$kind/200 (200 MB) -> $R" >> "$OUT"
|
||||
done
|
||||
echo "GET /stream/1 afterwards -> $(curl -s -o /dev/null -w 'status=%{http_code}' http://localhost:18080/stream/1)" >> "$OUT"
|
||||
kill $APP_PID; wait $APP_PID 2>/dev/null || true
|
||||
|
||||
OUT=cap/06-big-upload.txt; : > "$OUT"
|
||||
head -c 157286400 /dev/zero > /tmp/ft-150mb.bin
|
||||
run_app "--spring.servlet.multipart.max-file-size=-1 --spring.servlet.multipart.max-request-size=-1"
|
||||
echo "JVM started with -Xmx64m, multipart limits removed, uploading a 150 MB file" >> "$OUT"
|
||||
R=$(curl -s -o /dev/null -w "status=%{http_code} sent=%{size_upload} bytes in %{time_total}s" -F "file=@/tmp/ft-150mb.bin;type=application/octet-stream" http://localhost:18080/files || true)
|
||||
echo "multipart POST /files -> $R" >> "$OUT"
|
||||
R=$(curl -s -o /dev/null -w "status=%{http_code} sent=%{size_upload} bytes in %{time_total}s" -H "X-Filename: big.bin" --data-binary @/tmp/ft-150mb.bin -H "Content-Type: application/octet-stream" http://localhost:18080/files/raw || true)
|
||||
echo "raw POST /files/raw -> $R" >> "$OUT"
|
||||
kill $APP_PID; wait $APP_PID 2>/dev/null || true
|
||||
rm -f /tmp/ft-150mb.bin
|
||||
@@ -0,0 +1,42 @@
|
||||
package com.ankurm.filetransfer;
|
||||
|
||||
import java.nio.charset.StandardCharsets;
|
||||
|
||||
import org.springframework.core.io.FileSystemResource;
|
||||
import org.springframework.core.io.Resource;
|
||||
import org.springframework.http.ContentDisposition;
|
||||
import org.springframework.http.HttpHeaders;
|
||||
import org.springframework.http.HttpStatus;
|
||||
import org.springframework.http.MediaType;
|
||||
import org.springframework.http.ResponseEntity;
|
||||
import org.springframework.web.bind.annotation.GetMapping;
|
||||
import org.springframework.web.bind.annotation.PathVariable;
|
||||
import org.springframework.web.bind.annotation.RestController;
|
||||
import org.springframework.web.server.ResponseStatusException;
|
||||
|
||||
@RestController
|
||||
public class DownloadController {
|
||||
|
||||
private final FileStore store;
|
||||
|
||||
public DownloadController(FileStore store) {
|
||||
this.store = store;
|
||||
}
|
||||
|
||||
/**
|
||||
* Returning a Resource gets Range support for free: Spring answers "Range: bytes=..." with 206 Partial Content.
|
||||
* The content type is fixed to octet-stream on purpose: echoing back the type the client declared at upload
|
||||
* would let an uploader make the browser render their file as HTML.
|
||||
*/
|
||||
@GetMapping("/files/{id}")
|
||||
public ResponseEntity<Resource> download(@PathVariable String id) {
|
||||
FileStore.StoredFile f = store.find(id)
|
||||
.orElseThrow(() -> new ResponseStatusException(HttpStatus.NOT_FOUND, "no such file"));
|
||||
return ResponseEntity.ok()
|
||||
.header(HttpHeaders.CONTENT_DISPOSITION,
|
||||
ContentDisposition.attachment().filename(f.displayName(), StandardCharsets.UTF_8).build().toString())
|
||||
.header("X-Content-Type-Options", "nosniff")
|
||||
.contentType(MediaType.APPLICATION_OCTET_STREAM)
|
||||
.body(new FileSystemResource(f.path()));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,72 @@
|
||||
package com.ankurm.filetransfer;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.util.Map;
|
||||
import java.util.Optional;
|
||||
import java.util.UUID;
|
||||
import java.util.concurrent.ConcurrentHashMap;
|
||||
|
||||
import org.springframework.beans.factory.annotation.Value;
|
||||
import org.springframework.stereotype.Service;
|
||||
import org.springframework.web.multipart.MultipartFile;
|
||||
|
||||
/**
|
||||
* Stores uploads on local disk under a server-generated id. The client's filename is never used as a path:
|
||||
* it is kept only as display metadata, after stripping any directory part.
|
||||
*/
|
||||
@Service
|
||||
public class FileStore {
|
||||
|
||||
public record StoredFile(String id, String originalFilename, String displayName, long size,
|
||||
String declaredContentType, Path path) { }
|
||||
|
||||
private final Path root;
|
||||
private final Map<String, StoredFile> files = new ConcurrentHashMap<>();
|
||||
|
||||
public FileStore(@Value("${app.storage-dir:}") String dir) throws IOException {
|
||||
this.root = (dir.isBlank() ? Files.createTempDirectory("file-transfer") : Path.of(dir))
|
||||
.toAbsolutePath().normalize();
|
||||
Files.createDirectories(root);
|
||||
}
|
||||
|
||||
public Path root() {
|
||||
return root;
|
||||
}
|
||||
|
||||
public StoredFile save(MultipartFile file) throws IOException {
|
||||
String id = UUID.randomUUID().toString();
|
||||
Path target = root.resolve(id);
|
||||
file.transferTo(target);
|
||||
return remember(id, file.getOriginalFilename(), Files.size(target), file.getContentType(), target);
|
||||
}
|
||||
|
||||
public StoredFile saveStream(InputStream in, String name) throws IOException {
|
||||
String id = UUID.randomUUID().toString();
|
||||
Path target = root.resolve(id);
|
||||
long size = Files.copy(in, target);
|
||||
return remember(id, name, size, null, target);
|
||||
}
|
||||
|
||||
public Optional<StoredFile> find(String id) {
|
||||
return Optional.ofNullable(files.get(id));
|
||||
}
|
||||
|
||||
private StoredFile remember(String id, String original, long size, String contentType, Path path) {
|
||||
StoredFile stored = new StoredFile(id, original, displayName(original), size, contentType, path);
|
||||
files.put(id, stored);
|
||||
return stored;
|
||||
}
|
||||
|
||||
/** Last path segment only, control characters removed. Handles both / and \ separators. */
|
||||
static String displayName(String raw) {
|
||||
if (raw == null || raw.isBlank()) {
|
||||
return "unnamed";
|
||||
}
|
||||
String name = raw.replace('\\', '/');
|
||||
name = name.substring(name.lastIndexOf('/') + 1).replaceAll("\\p{Cntrl}", "");
|
||||
return name.isBlank() ? "unnamed" : name;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
package com.ankurm.filetransfer;
|
||||
|
||||
import org.springframework.boot.SpringApplication;
|
||||
import org.springframework.boot.autoconfigure.SpringBootApplication;
|
||||
|
||||
@SpringBootApplication
|
||||
public class FileTransferApplication {
|
||||
|
||||
public static void main(String[] args) {
|
||||
SpringApplication.run(FileTransferApplication.class, args);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,42 @@
|
||||
package com.ankurm.filetransfer;
|
||||
|
||||
import java.time.Duration;
|
||||
import java.util.Map;
|
||||
import java.util.Set;
|
||||
|
||||
import org.springframework.http.HttpStatus;
|
||||
import org.springframework.web.bind.annotation.GetMapping;
|
||||
import org.springframework.web.bind.annotation.PathVariable;
|
||||
import org.springframework.web.bind.annotation.PostMapping;
|
||||
import org.springframework.web.bind.annotation.RequestBody;
|
||||
import org.springframework.web.bind.annotation.RequestParam;
|
||||
import org.springframework.web.bind.annotation.RestController;
|
||||
import org.springframework.web.server.ResponseStatusException;
|
||||
|
||||
@RestController
|
||||
public class PresignController {
|
||||
|
||||
public record UploadRequest(String contentType) { }
|
||||
|
||||
private static final Set<String> ALLOWED = Set.of("image/png", "image/jpeg", "application/pdf");
|
||||
private static final Duration TTL = Duration.ofMinutes(5);
|
||||
|
||||
private final S3PresignService presign;
|
||||
|
||||
public PresignController(S3PresignService presign) {
|
||||
this.presign = presign;
|
||||
}
|
||||
|
||||
@PostMapping("/presign/upload")
|
||||
public S3PresignService.Upload upload(@RequestBody UploadRequest request) {
|
||||
if (request.contentType() == null || !ALLOWED.contains(request.contentType())) {
|
||||
throw new ResponseStatusException(HttpStatus.BAD_REQUEST, "content type not allowed");
|
||||
}
|
||||
return presign.forUpload(request.contentType(), TTL);
|
||||
}
|
||||
|
||||
@GetMapping("/presign/download/{key}")
|
||||
public Map<String, String> download(@PathVariable String key, @RequestParam(defaultValue = "download.bin") String filename) {
|
||||
return Map.of("url", presign.forDownload(key, filename, TTL));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,68 @@
|
||||
package com.ankurm.filetransfer;
|
||||
|
||||
import java.net.URI;
|
||||
import java.time.Duration;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.UUID;
|
||||
|
||||
import org.springframework.beans.factory.annotation.Value;
|
||||
import org.springframework.stereotype.Service;
|
||||
|
||||
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
|
||||
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
|
||||
import software.amazon.awssdk.regions.Region;
|
||||
import software.amazon.awssdk.services.s3.S3Configuration;
|
||||
import software.amazon.awssdk.services.s3.model.GetObjectRequest;
|
||||
import software.amazon.awssdk.services.s3.model.PutObjectRequest;
|
||||
import software.amazon.awssdk.services.s3.presigner.S3Presigner;
|
||||
import software.amazon.awssdk.services.s3.presigner.model.PresignedGetObjectRequest;
|
||||
import software.amazon.awssdk.services.s3.presigner.model.PresignedPutObjectRequest;
|
||||
|
||||
/** Signs S3 / MinIO URLs locally. Nothing here talks to the network. */
|
||||
@Service
|
||||
public class S3PresignService implements AutoCloseable {
|
||||
|
||||
public record Upload(String key, String url, String method, Map<String, String> headers, long expiresInSeconds) { }
|
||||
|
||||
private final S3Presigner presigner;
|
||||
private final String bucket;
|
||||
|
||||
public S3PresignService(@Value("${app.s3.endpoint}") String endpoint, @Value("${app.s3.bucket}") String bucket,
|
||||
@Value("${app.s3.access-key}") String accessKey, @Value("${app.s3.secret-key}") String secretKey) {
|
||||
this.bucket = bucket;
|
||||
this.presigner = S3Presigner.builder()
|
||||
.region(Region.US_EAST_1)
|
||||
.endpointOverride(URI.create(endpoint))
|
||||
.credentialsProvider(StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey)))
|
||||
.serviceConfiguration(S3Configuration.builder().pathStyleAccessEnabled(true).build())
|
||||
.build();
|
||||
}
|
||||
|
||||
/** The object key is chosen here, never by the client. The content type is part of the signature. */
|
||||
public Upload forUpload(String contentType, Duration ttl) {
|
||||
String key = UUID.randomUUID().toString();
|
||||
PutObjectRequest put = PutObjectRequest.builder().bucket(bucket).key(key).contentType(contentType).build();
|
||||
PresignedPutObjectRequest signed = presigner.presignPutObject(r -> r.signatureDuration(ttl).putObjectRequest(put));
|
||||
Map<String, String> headers = new LinkedHashMap<>();
|
||||
for (Map.Entry<String, List<String>> h : signed.signedHeaders().entrySet()) {
|
||||
if (!h.getKey().equalsIgnoreCase("host")) {
|
||||
headers.put(h.getKey(), String.join(",", h.getValue()));
|
||||
}
|
||||
}
|
||||
return new Upload(key, signed.url().toString(), "PUT", headers, ttl.toSeconds());
|
||||
}
|
||||
|
||||
public String forDownload(String key, String filename, Duration ttl) {
|
||||
GetObjectRequest get = GetObjectRequest.builder().bucket(bucket).key(key)
|
||||
.responseContentDisposition("attachment; filename=\"" + filename.replace("\"", "") + "\"").build();
|
||||
PresignedGetObjectRequest signed = presigner.presignGetObject(r -> r.signatureDuration(ttl).getObjectRequest(get));
|
||||
return signed.url().toString();
|
||||
}
|
||||
|
||||
@Override
|
||||
public void close() {
|
||||
presigner.close();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,39 @@
|
||||
package com.ankurm.filetransfer;
|
||||
|
||||
import java.util.Arrays;
|
||||
|
||||
import org.springframework.http.MediaType;
|
||||
import org.springframework.http.ResponseEntity;
|
||||
import org.springframework.web.bind.annotation.GetMapping;
|
||||
import org.springframework.web.bind.annotation.PathVariable;
|
||||
import org.springframework.web.bind.annotation.RestController;
|
||||
import org.springframework.web.servlet.mvc.method.annotation.StreamingResponseBody;
|
||||
|
||||
/** The same N megabytes of data, sent two ways. Run both under a small heap and compare. */
|
||||
@RestController
|
||||
public class StreamController {
|
||||
|
||||
private static final int CHUNK = 64 * 1024;
|
||||
|
||||
/** Streams in 64 KB chunks: memory use does not depend on the size of the file. */
|
||||
@GetMapping("/stream/{mb}")
|
||||
public ResponseEntity<StreamingResponseBody> stream(@PathVariable int mb) {
|
||||
long total = (long) mb * 1024 * 1024;
|
||||
StreamingResponseBody body = out -> {
|
||||
byte[] chunk = new byte[CHUNK];
|
||||
Arrays.fill(chunk, (byte) 'x');
|
||||
for (long written = 0; written < total; written += CHUNK) {
|
||||
out.write(chunk, 0, (int) Math.min(CHUNK, total - written));
|
||||
}
|
||||
};
|
||||
return ResponseEntity.ok().contentType(MediaType.APPLICATION_OCTET_STREAM).contentLength(total).body(body);
|
||||
}
|
||||
|
||||
/** Builds the whole thing in memory first: the heap must hold the entire file. */
|
||||
@GetMapping(path = "/buffered/{mb}", produces = MediaType.APPLICATION_OCTET_STREAM_VALUE)
|
||||
public byte[] buffered(@PathVariable int mb) {
|
||||
byte[] all = new byte[mb * 1024 * 1024];
|
||||
Arrays.fill(all, (byte) 'x');
|
||||
return all;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,49 @@
|
||||
package com.ankurm.filetransfer;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.net.URI;
|
||||
|
||||
import org.springframework.http.HttpStatus;
|
||||
import org.springframework.http.MediaType;
|
||||
import org.springframework.http.ResponseEntity;
|
||||
import org.springframework.web.bind.annotation.PostMapping;
|
||||
import org.springframework.web.bind.annotation.RequestHeader;
|
||||
import org.springframework.web.bind.annotation.RequestParam;
|
||||
import org.springframework.web.bind.annotation.RestController;
|
||||
import org.springframework.web.multipart.MultipartFile;
|
||||
import org.springframework.web.server.ResponseStatusException;
|
||||
|
||||
@RestController
|
||||
public class UploadController {
|
||||
|
||||
public record UploadResult(String id, String originalFilename, String displayName, long size,
|
||||
String declaredContentType) { }
|
||||
|
||||
private final FileStore store;
|
||||
|
||||
public UploadController(FileStore store) {
|
||||
this.store = store;
|
||||
}
|
||||
|
||||
/** Multipart upload: the container parses the body and spills big parts to a temp file. */
|
||||
@PostMapping(path = "/files", consumes = MediaType.MULTIPART_FORM_DATA_VALUE)
|
||||
public ResponseEntity<UploadResult> upload(@RequestParam("file") MultipartFile file) throws IOException {
|
||||
if (file.isEmpty()) {
|
||||
throw new ResponseStatusException(HttpStatus.BAD_REQUEST, "empty file");
|
||||
}
|
||||
return created(store.save(file));
|
||||
}
|
||||
|
||||
/** Raw upload: the request body is the file. No multipart parsing, so the multipart limits do not apply. */
|
||||
@PostMapping("/files/raw")
|
||||
public ResponseEntity<UploadResult> raw(InputStream body,
|
||||
@RequestHeader(value = "X-Filename", defaultValue = "upload.bin") String name) throws IOException {
|
||||
return created(store.saveStream(body, name));
|
||||
}
|
||||
|
||||
private static ResponseEntity<UploadResult> created(FileStore.StoredFile s) {
|
||||
return ResponseEntity.created(URI.create("/files/" + s.id()))
|
||||
.body(new UploadResult(s.id(), s.originalFilename(), s.displayName(), s.size(), s.declaredContentType()));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
package com.ankurm.filetransfer;
|
||||
|
||||
import org.springframework.http.HttpStatus;
|
||||
import org.springframework.http.ProblemDetail;
|
||||
import org.springframework.web.bind.annotation.ExceptionHandler;
|
||||
import org.springframework.web.bind.annotation.RestControllerAdvice;
|
||||
import org.springframework.web.multipart.MaxUploadSizeExceededException;
|
||||
|
||||
/** Turns the container's size-limit failure into a JSON problem instead of an empty 413. */
|
||||
@RestControllerAdvice
|
||||
public class UploadErrors {
|
||||
|
||||
@ExceptionHandler(MaxUploadSizeExceededException.class)
|
||||
public ProblemDetail tooLarge(MaxUploadSizeExceededException e) {
|
||||
ProblemDetail pd = ProblemDetail.forStatusAndDetail(HttpStatus.CONTENT_TOO_LARGE,
|
||||
"The upload is larger than the configured limit (spring.servlet.multipart.max-file-size / max-request-size).");
|
||||
pd.setTitle("Upload too large");
|
||||
return pd;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
# Per-file and per-request limits. Enforced by the servlet container (Tomcat), not by Spring MVC.
|
||||
spring.servlet.multipart.max-file-size=1MB
|
||||
spring.servlet.multipart.max-request-size=2MB
|
||||
|
||||
# Where uploads land. Blank = a fresh temp directory per run.
|
||||
app.storage-dir=
|
||||
|
||||
# S3 / MinIO settings for the presigner. These are the example keys from the AWS documentation,
|
||||
# not real credentials. Override all four for a real bucket.
|
||||
app.s3.endpoint=http://localhost:9000
|
||||
app.s3.bucket=uploads
|
||||
app.s3.access-key=AKIAIOSFODNN7EXAMPLE
|
||||
app.s3.secret-key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
|
||||
@@ -0,0 +1,89 @@
|
||||
package com.ankurm.filetransfer;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.io.UncheckedIOException;
|
||||
import java.net.URI;
|
||||
import java.net.http.HttpClient;
|
||||
import java.net.http.HttpRequest;
|
||||
import java.net.http.HttpResponse;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.nio.file.StandardOpenOption;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
|
||||
/** Test helpers: write the transcript to cap/, and build multipart bodies by hand so the test controls every byte. */
|
||||
final class Cap {
|
||||
|
||||
static final Path DIR = Path.of("cap");
|
||||
static final HttpClient HTTP = HttpClient.newHttpClient();
|
||||
|
||||
private Cap() { }
|
||||
|
||||
static void reset(String file) {
|
||||
try {
|
||||
Files.createDirectories(DIR);
|
||||
Files.deleteIfExists(DIR.resolve(file));
|
||||
} catch (IOException e) {
|
||||
throw new UncheckedIOException(e);
|
||||
}
|
||||
}
|
||||
|
||||
static void say(String file, String line) {
|
||||
System.out.println(line);
|
||||
try {
|
||||
Files.writeString(DIR.resolve(file), line + System.lineSeparator(),
|
||||
StandardOpenOption.CREATE, StandardOpenOption.APPEND);
|
||||
} catch (IOException e) {
|
||||
throw new UncheckedIOException(e);
|
||||
}
|
||||
}
|
||||
|
||||
record Part(String field, String filename, String contentType, byte[] content) { }
|
||||
|
||||
static byte[] multipart(String boundary, List<Part> parts) {
|
||||
List<byte[]> chunks = new ArrayList<>();
|
||||
for (Part p : parts) {
|
||||
String head = "--" + boundary + "\r\n"
|
||||
+ "Content-Disposition: form-data; name=\"" + p.field() + "\"; filename=\"" + p.filename() + "\"\r\n"
|
||||
+ "Content-Type: " + p.contentType() + "\r\n\r\n";
|
||||
chunks.add(head.getBytes(StandardCharsets.UTF_8));
|
||||
chunks.add(p.content());
|
||||
chunks.add("\r\n".getBytes(StandardCharsets.UTF_8));
|
||||
}
|
||||
chunks.add(("--" + boundary + "--\r\n").getBytes(StandardCharsets.UTF_8));
|
||||
int len = chunks.stream().mapToInt(c -> c.length).sum();
|
||||
byte[] all = new byte[len];
|
||||
int pos = 0;
|
||||
for (byte[] c : chunks) {
|
||||
System.arraycopy(c, 0, all, pos, c.length);
|
||||
pos += c.length;
|
||||
}
|
||||
return all;
|
||||
}
|
||||
|
||||
/** POST the parts to /files; returns "status=NNN body=..." or the exception if the connection broke. */
|
||||
static String postMultipart(String base, List<Part> parts) {
|
||||
String boundary = "----ft" + System.nanoTime();
|
||||
HttpRequest req = HttpRequest.newBuilder(URI.create(base + "/files"))
|
||||
.header("Content-Type", "multipart/form-data; boundary=" + boundary)
|
||||
.POST(HttpRequest.BodyPublishers.ofByteArray(multipart(boundary, parts))).build();
|
||||
try {
|
||||
HttpResponse<String> r = HTTP.send(req, HttpResponse.BodyHandlers.ofString());
|
||||
String body = r.body().replaceAll("\\s+", " ");
|
||||
return "status=" + r.statusCode() + " body=" + (body.length() > 140 ? body.substring(0, 140) + "..." : body);
|
||||
} catch (IOException e) {
|
||||
return "no response, " + e.getClass().getSimpleName() + ": " + e.getMessage();
|
||||
} catch (InterruptedException e) {
|
||||
Thread.currentThread().interrupt();
|
||||
return "interrupted";
|
||||
}
|
||||
}
|
||||
|
||||
static byte[] bytes(int n) {
|
||||
byte[] b = new byte[n];
|
||||
java.util.Arrays.fill(b, (byte) 'a');
|
||||
return b;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,164 @@
|
||||
package com.ankurm.filetransfer;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.net.InetSocketAddress;
|
||||
import java.net.URLDecoder;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.security.MessageDigest;
|
||||
import java.time.Instant;
|
||||
import java.time.LocalDateTime;
|
||||
import java.time.ZoneOffset;
|
||||
import java.time.format.DateTimeFormatter;
|
||||
import java.util.ArrayList;
|
||||
import java.util.HexFormat;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.TreeMap;
|
||||
import java.util.concurrent.ConcurrentHashMap;
|
||||
|
||||
import javax.crypto.Mac;
|
||||
import javax.crypto.spec.SecretKeySpec;
|
||||
|
||||
import com.sun.net.httpserver.HttpExchange;
|
||||
import com.sun.net.httpserver.HttpServer;
|
||||
|
||||
/**
|
||||
* A tiny stand-in for S3 / MinIO that accepts PUT and GET on /{bucket}/{key} and checks the AWS Signature V4
|
||||
* query-string signature, following the published algorithm. It exists because this build environment has no
|
||||
* Docker. It proves the presigned URLs are self-consistent with the algorithm as documented; it does not prove
|
||||
* that a real S3 or MinIO server would accept them.
|
||||
*/
|
||||
final class FakeS3 implements AutoCloseable {
|
||||
|
||||
record Obj(byte[] data, String contentType) { }
|
||||
|
||||
final Map<String, Obj> objects = new ConcurrentHashMap<>();
|
||||
private final HttpServer server;
|
||||
private final String secretKey;
|
||||
|
||||
FakeS3(String secretKey) throws IOException {
|
||||
this.secretKey = secretKey;
|
||||
this.server = HttpServer.create(new InetSocketAddress("localhost", 0), 0);
|
||||
this.server.createContext("/", this::handle);
|
||||
this.server.start();
|
||||
}
|
||||
|
||||
String endpoint() {
|
||||
return "http://localhost:" + server.getAddress().getPort();
|
||||
}
|
||||
|
||||
@Override
|
||||
public void close() {
|
||||
server.stop(0);
|
||||
}
|
||||
|
||||
private void handle(HttpExchange ex) throws IOException {
|
||||
try {
|
||||
String error = verify(ex);
|
||||
if (error != null) {
|
||||
String[] parts = error.split("\\|", 2);
|
||||
reply(ex, 403, "<Error><Code>" + parts[0] + "</Code><Message>" + parts[1] + "</Message></Error>", "application/xml", null);
|
||||
return;
|
||||
}
|
||||
String path = ex.getRequestURI().getRawPath();
|
||||
if (ex.getRequestMethod().equals("PUT")) {
|
||||
objects.put(path, new Obj(ex.getRequestBody().readAllBytes(), ex.getRequestHeaders().getFirst("Content-Type")));
|
||||
reply(ex, 200, "", "text/plain", null);
|
||||
} else if (ex.getRequestMethod().equals("GET")) {
|
||||
Obj o = objects.get(path);
|
||||
if (o == null) {
|
||||
reply(ex, 404, "<Error><Code>NoSuchKey</Code></Error>", "application/xml", null);
|
||||
return;
|
||||
}
|
||||
String cd = queryParams(ex).get("response-content-disposition");
|
||||
reply(ex, 200, o.data(), o.contentType() == null ? "application/octet-stream" : o.contentType(),
|
||||
cd == null ? null : URLDecoder.decode(cd, StandardCharsets.UTF_8));
|
||||
} else {
|
||||
reply(ex, 405, "", "text/plain", null);
|
||||
}
|
||||
} catch (Exception e) {
|
||||
reply(ex, 500, e.toString(), "text/plain", null);
|
||||
}
|
||||
}
|
||||
|
||||
private static Map<String, String> queryParams(HttpExchange ex) {
|
||||
Map<String, String> m = new TreeMap<>();
|
||||
String raw = ex.getRequestURI().getRawQuery();
|
||||
if (raw != null) {
|
||||
for (String kv : raw.split("&")) {
|
||||
int i = kv.indexOf('=');
|
||||
m.put(i < 0 ? kv : kv.substring(0, i), i < 0 ? "" : kv.substring(i + 1));
|
||||
}
|
||||
}
|
||||
return m;
|
||||
}
|
||||
|
||||
/** Returns null if the signature is valid, otherwise "Code|Message". */
|
||||
private String verify(HttpExchange ex) throws Exception {
|
||||
Map<String, String> q = queryParams(ex);
|
||||
String signature = q.get("X-Amz-Signature");
|
||||
if (signature == null || !"AWS4-HMAC-SHA256".equals(q.get("X-Amz-Algorithm"))) {
|
||||
return "AccessDenied|Query-string authentication requires the X-Amz-Algorithm and X-Amz-Signature parameters";
|
||||
}
|
||||
String amzDate = q.get("X-Amz-Date");
|
||||
long expires = Long.parseLong(q.get("X-Amz-Expires"));
|
||||
Instant signedAt = LocalDateTime.parse(amzDate, DateTimeFormatter.ofPattern("yyyyMMdd'T'HHmmss'Z'")).toInstant(ZoneOffset.UTC);
|
||||
if (Instant.now().isAfter(signedAt.plusSeconds(expires))) {
|
||||
return "AccessDenied|Request has expired";
|
||||
}
|
||||
|
||||
String credential = URLDecoder.decode(q.get("X-Amz-Credential"), StandardCharsets.UTF_8);
|
||||
String[] scope = credential.split("/"); // accessKey/date/region/service/aws4_request
|
||||
String signedHeaders = URLDecoder.decode(q.get("X-Amz-SignedHeaders"), StandardCharsets.UTF_8);
|
||||
|
||||
Map<String, String> signedQuery = new TreeMap<>(q);
|
||||
signedQuery.remove("X-Amz-Signature");
|
||||
List<String> pairs = new ArrayList<>();
|
||||
signedQuery.forEach((k, v) -> pairs.add(k + "=" + v));
|
||||
|
||||
StringBuilder canonicalHeaders = new StringBuilder();
|
||||
for (String h : signedHeaders.split(";")) {
|
||||
String value = h.equals("host") ? ex.getRequestHeaders().getFirst("Host") : ex.getRequestHeaders().getFirst(h);
|
||||
canonicalHeaders.append(h).append(':').append(value == null ? "" : value.trim()).append('\n');
|
||||
}
|
||||
|
||||
String canonicalRequest = ex.getRequestMethod() + "\n" + ex.getRequestURI().getRawPath() + "\n"
|
||||
+ String.join("&", pairs) + "\n" + canonicalHeaders + "\n" + signedHeaders + "\nUNSIGNED-PAYLOAD";
|
||||
String stringToSign = "AWS4-HMAC-SHA256\n" + amzDate + "\n" + scope[1] + "/" + scope[2] + "/" + scope[3] + "/aws4_request\n"
|
||||
+ hex(MessageDigest.getInstance("SHA-256").digest(canonicalRequest.getBytes(StandardCharsets.UTF_8)));
|
||||
|
||||
byte[] k = hmac(("AWS4" + secretKey).getBytes(StandardCharsets.UTF_8), scope[1]);
|
||||
k = hmac(k, scope[2]);
|
||||
k = hmac(k, scope[3]);
|
||||
k = hmac(k, "aws4_request");
|
||||
String expected = hex(hmac(k, stringToSign));
|
||||
return expected.equals(signature) ? null
|
||||
: "SignatureDoesNotMatch|The request signature we calculated does not match the signature you provided";
|
||||
}
|
||||
|
||||
private static byte[] hmac(byte[] key, String data) throws Exception {
|
||||
Mac mac = Mac.getInstance("HmacSHA256");
|
||||
mac.init(new SecretKeySpec(key, "HmacSHA256"));
|
||||
return mac.doFinal(data.getBytes(StandardCharsets.UTF_8));
|
||||
}
|
||||
|
||||
private static String hex(byte[] b) {
|
||||
return HexFormat.of().formatHex(b);
|
||||
}
|
||||
|
||||
private static void reply(HttpExchange ex, int status, String body, String type, String disposition) throws IOException {
|
||||
reply(ex, status, body.getBytes(StandardCharsets.UTF_8), type, disposition);
|
||||
}
|
||||
|
||||
private static void reply(HttpExchange ex, int status, byte[] body, String type, String disposition) throws IOException {
|
||||
ex.getResponseHeaders().set("Content-Type", type);
|
||||
if (disposition != null) {
|
||||
ex.getResponseHeaders().set("Content-Disposition", disposition);
|
||||
}
|
||||
ex.sendResponseHeaders(status, body.length == 0 ? -1 : body.length);
|
||||
if (body.length > 0) {
|
||||
ex.getResponseBody().write(body);
|
||||
}
|
||||
ex.close();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,92 @@
|
||||
package com.ankurm.filetransfer;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
import java.net.URI;
|
||||
import java.net.http.HttpRequest;
|
||||
import java.net.http.HttpResponse;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.nio.file.Path;
|
||||
import java.util.List;
|
||||
|
||||
import org.junit.jupiter.api.BeforeAll;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.boot.test.context.SpringBootTest;
|
||||
import org.springframework.boot.test.web.server.LocalServerPort;
|
||||
|
||||
import com.ankurm.filetransfer.Cap.Part;
|
||||
|
||||
@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
|
||||
class FilenameAndDownloadTest {
|
||||
|
||||
static final String F = "02-filename.txt";
|
||||
static final String G = "03-download.txt";
|
||||
|
||||
@LocalServerPort int port;
|
||||
@Autowired FileStore store;
|
||||
|
||||
@BeforeAll
|
||||
static void clean() {
|
||||
Cap.reset(F);
|
||||
Cap.reset(G);
|
||||
}
|
||||
|
||||
String base() {
|
||||
return "http://localhost:" + port;
|
||||
}
|
||||
|
||||
String postRaw(String filename, String contentType, byte[] content) throws Exception {
|
||||
String boundary = "----ft" + System.nanoTime();
|
||||
byte[] body = Cap.multipart(boundary, List.of(new Part("file", filename, contentType, content)));
|
||||
HttpRequest req = HttpRequest.newBuilder(URI.create(base() + "/files"))
|
||||
.header("Content-Type", "multipart/form-data; boundary=" + boundary)
|
||||
.POST(HttpRequest.BodyPublishers.ofByteArray(body)).build();
|
||||
return Cap.HTTP.send(req, HttpResponse.BodyHandlers.ofString()).body();
|
||||
}
|
||||
|
||||
@Test
|
||||
void whatTheServerSeesIsWhatTheClientSent() throws Exception {
|
||||
String evil = "../../etc/cron.d/evil.txt";
|
||||
String json = postRaw(evil, "image/png", "hello".getBytes(StandardCharsets.UTF_8));
|
||||
Cap.say(F, "client sent filename=" + evil + " Content-Type=image/png");
|
||||
Cap.say(F, "server response: " + json);
|
||||
|
||||
Path root = store.root();
|
||||
boolean escapes = !root.resolve(evil).normalize().startsWith(root);
|
||||
Cap.say(F, "root.resolve(originalFilename).normalize() stays inside the storage dir: " + !escapes);
|
||||
assertThat(json).contains("\"displayName\":\"evil.txt\"");
|
||||
assertThat(escapes).isTrue();
|
||||
|
||||
String win = postRaw("..\\..\\windows\\system32\\evil.dll", "text/plain", "x".getBytes(StandardCharsets.UTF_8));
|
||||
Cap.say(F, "client sent a Windows-style path; server response: " + win);
|
||||
}
|
||||
|
||||
@Test
|
||||
void downloadSupportsRangeAndEncodesNames() throws Exception {
|
||||
String json = postRaw("résumé 2026.txt", "text/plain", "ABCDEFGHIJKLMNOPQRSTUVWXYZ".getBytes(StandardCharsets.UTF_8));
|
||||
String id = json.replaceAll(".*\"id\":\"([^\"]+)\".*", "$1");
|
||||
|
||||
HttpResponse<String> full = Cap.HTTP.send(HttpRequest.newBuilder(URI.create(base() + "/files/" + id)).build(),
|
||||
HttpResponse.BodyHandlers.ofString());
|
||||
Cap.say(G, "GET /files/{id}");
|
||||
Cap.say(G, " status=" + full.statusCode() + " body=" + full.body());
|
||||
for (String h : List.of("content-type", "content-length", "content-disposition", "accept-ranges", "x-content-type-options")) {
|
||||
Cap.say(G, " " + h + ": " + full.headers().firstValue(h).orElse("(absent)"));
|
||||
}
|
||||
|
||||
HttpResponse<String> part = Cap.HTTP.send(HttpRequest.newBuilder(URI.create(base() + "/files/" + id))
|
||||
.header("Range", "bytes=10-19").build(), HttpResponse.BodyHandlers.ofString());
|
||||
Cap.say(G, "GET /files/{id} with Range: bytes=10-19");
|
||||
Cap.say(G, " status=" + part.statusCode() + " body=" + part.body());
|
||||
Cap.say(G, " content-range: " + part.headers().firstValue("content-range").orElse("(absent)"));
|
||||
|
||||
HttpResponse<String> past = Cap.HTTP.send(HttpRequest.newBuilder(URI.create(base() + "/files/" + id))
|
||||
.header("Range", "bytes=500-").build(), HttpResponse.BodyHandlers.ofString());
|
||||
Cap.say(G, "GET /files/{id} with Range: bytes=500-");
|
||||
Cap.say(G, " status=" + past.statusCode() + " content-range: " + past.headers().firstValue("content-range").orElse("(absent)"));
|
||||
|
||||
assertThat(part.statusCode()).isEqualTo(206);
|
||||
assertThat(part.body()).isEqualTo("KLMNOPQRST");
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,74 @@
|
||||
package com.ankurm.filetransfer;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
import org.junit.jupiter.api.BeforeAll;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.boot.test.context.SpringBootTest;
|
||||
import org.springframework.boot.test.web.server.LocalServerPort;
|
||||
import org.springframework.boot.webmvc.test.autoconfigure.AutoConfigureMockMvc;
|
||||
import org.springframework.mock.web.MockMultipartFile;
|
||||
import org.springframework.test.web.servlet.MockMvc;
|
||||
import org.springframework.test.web.servlet.request.MockMvcRequestBuilders;
|
||||
|
||||
import com.ankurm.filetransfer.Cap.Part;
|
||||
|
||||
@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
|
||||
@AutoConfigureMockMvc
|
||||
class MultipartLimitsTest {
|
||||
|
||||
static final String F = "01-limits.txt";
|
||||
static final int KB = 1024;
|
||||
|
||||
@LocalServerPort int port;
|
||||
@Autowired MockMvc mvc;
|
||||
|
||||
@BeforeAll
|
||||
static void clean() {
|
||||
Cap.reset(F);
|
||||
}
|
||||
|
||||
String base() {
|
||||
return "http://localhost:" + port;
|
||||
}
|
||||
|
||||
@Test
|
||||
void limitsAreEnforcedByTheContainerNotByMockMvc() throws Exception {
|
||||
Cap.say(F, "limits: max-file-size=1MB, max-request-size=2MB");
|
||||
|
||||
String small = Cap.postMultipart(base(), List.of(new Part("file", "a.txt", "text/plain", Cap.bytes(500 * KB))));
|
||||
Cap.say(F, "real HTTP, one 500 KB file -> " + small);
|
||||
|
||||
String big = Cap.postMultipart(base(), List.of(new Part("file", "b.txt", "text/plain", Cap.bytes(1536 * KB))));
|
||||
Cap.say(F, "real HTTP, one 1.5 MB file -> " + big);
|
||||
|
||||
List<Part> three = List.of(
|
||||
new Part("file", "c1.txt", "text/plain", Cap.bytes(900 * KB)),
|
||||
new Part("file", "c2.txt", "text/plain", Cap.bytes(900 * KB)),
|
||||
new Part("file", "c3.txt", "text/plain", Cap.bytes(900 * KB)));
|
||||
String request = Cap.postMultipart(base(), three);
|
||||
Cap.say(F, "real HTTP, 3 x 900 KB in one request -> " + request);
|
||||
|
||||
// Tomcat only swallows a limited amount (maxSwallowSize, 2 MB) of a rejected upload before it closes the
|
||||
// connection, so a client sending far more than that may see a reset instead of the 413. Count outcomes.
|
||||
int got413 = 0, reset = 0;
|
||||
for (int i = 0; i < 10; i++) {
|
||||
String huge = Cap.postMultipart(base(), List.of(new Part("file", "d.bin", "application/octet-stream", Cap.bytes(10 * 1024 * KB))));
|
||||
if (huge.startsWith("status=413")) got413++;
|
||||
else reset++;
|
||||
}
|
||||
Cap.say(F, "real HTTP, one 10 MB file, 10 attempts -> " + got413 + " x status=413, " + reset + " x connection error");
|
||||
|
||||
int mockStatus = mvc.perform(MockMvcRequestBuilders.multipart("/files")
|
||||
.file(new MockMultipartFile("file", "b.txt", "text/plain", Cap.bytes(1536 * KB))))
|
||||
.andReturn().getResponse().getStatus();
|
||||
Cap.say(F, "MockMvc, the same 1.5 MB file -> status=" + mockStatus);
|
||||
|
||||
assertThat(small).startsWith("status=201");
|
||||
assertThat(big).startsWith("status=413");
|
||||
assertThat(mockStatus).isEqualTo(201);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,118 @@
|
||||
package com.ankurm.filetransfer;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.net.URI;
|
||||
import java.net.http.HttpRequest;
|
||||
import java.net.http.HttpResponse;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.time.Duration;
|
||||
import java.util.Arrays;
|
||||
|
||||
import org.junit.jupiter.api.AfterAll;
|
||||
import org.junit.jupiter.api.BeforeAll;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.boot.test.context.SpringBootTest;
|
||||
import org.springframework.boot.test.web.server.LocalServerPort;
|
||||
import org.springframework.test.context.DynamicPropertyRegistry;
|
||||
import org.springframework.test.context.DynamicPropertySource;
|
||||
|
||||
import tools.jackson.databind.JsonNode;
|
||||
import tools.jackson.databind.json.JsonMapper;
|
||||
|
||||
@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
|
||||
class PresignedUrlTest {
|
||||
|
||||
static final String F = "04-presign.txt";
|
||||
static final String SECRET = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY";
|
||||
static FakeS3 s3;
|
||||
|
||||
@LocalServerPort int port;
|
||||
@Autowired S3PresignService presign;
|
||||
JsonMapper json = JsonMapper.builder().build();
|
||||
|
||||
@DynamicPropertySource
|
||||
static void s3(DynamicPropertyRegistry r) throws IOException {
|
||||
s3 = new FakeS3(SECRET);
|
||||
r.add("app.s3.endpoint", s3::endpoint);
|
||||
}
|
||||
|
||||
@BeforeAll
|
||||
static void clean() {
|
||||
Cap.reset(F);
|
||||
}
|
||||
|
||||
@AfterAll
|
||||
static void stop() {
|
||||
s3.close();
|
||||
}
|
||||
|
||||
HttpResponse<String> put(String url, String contentType, byte[] body) throws Exception {
|
||||
HttpRequest.Builder b = HttpRequest.newBuilder(URI.create(url)).PUT(HttpRequest.BodyPublishers.ofByteArray(body));
|
||||
if (contentType != null) {
|
||||
b.header("Content-Type", contentType);
|
||||
}
|
||||
return Cap.HTTP.send(b.build(), HttpResponse.BodyHandlers.ofString());
|
||||
}
|
||||
|
||||
static String brief(HttpResponse<String> r) {
|
||||
String body = r.body().replaceAll("<Message>.*</Message>", "").replaceAll("\\s+", " ");
|
||||
return "status=" + r.statusCode() + (body.isBlank() ? "" : " " + body);
|
||||
}
|
||||
|
||||
@Test
|
||||
void presignedUploadAndDownload() throws Exception {
|
||||
HttpResponse<String> res = Cap.HTTP.send(HttpRequest.newBuilder(URI.create("http://localhost:" + port + "/presign/upload"))
|
||||
.header("Content-Type", "application/json")
|
||||
.POST(HttpRequest.BodyPublishers.ofString("{\"contentType\":\"image/png\"}")).build(), HttpResponse.BodyHandlers.ofString());
|
||||
JsonNode node = json.readTree(res.body());
|
||||
String url = node.get("url").asString();
|
||||
String key = node.get("key").asString();
|
||||
|
||||
Cap.say(F, "POST /presign/upload {\"contentType\":\"image/png\"} -> status=" + res.statusCode());
|
||||
Cap.say(F, " method=" + node.get("method").asString() + " expiresInSeconds=" + node.get("expiresInSeconds").asLong()
|
||||
+ " headers=" + node.get("headers"));
|
||||
String[] split = url.split("\\?");
|
||||
Cap.say(F, " url path: " + split[0].replaceFirst("http://localhost:\\d+", "http://<fake-s3>"));
|
||||
for (String p : split[1].split("&")) {
|
||||
Cap.say(F, " query: " + p);
|
||||
}
|
||||
|
||||
byte[] png = {(byte) 0x89, 'P', 'N', 'G', 1, 2, 3, 4};
|
||||
Cap.say(F, "PUT with the signed Content-Type (image/png) -> " + brief(put(url, "image/png", png)));
|
||||
assertThat(s3.objects.values().iterator().next().data()).isEqualTo(png);
|
||||
|
||||
Cap.say(F, "PUT the same URL again (presigned URLs are reusable) -> " + brief(put(url, "image/png", png)));
|
||||
Cap.say(F, "PUT with a different Content-Type (text/html) -> " + brief(put(url, "text/html", png)));
|
||||
Cap.say(F, "PUT with no Content-Type header -> " + brief(put(url, null, png)));
|
||||
|
||||
String tampered = url.replace(key, key.substring(0, key.length() - 1) + (key.endsWith("a") ? "b" : "a"));
|
||||
Cap.say(F, "PUT after changing the last character of the key -> " + brief(put(tampered, "image/png", png)));
|
||||
|
||||
String longer = url.replace("X-Amz-Expires=300", "X-Amz-Expires=604800");
|
||||
Cap.say(F, "PUT after raising X-Amz-Expires to 7 days -> " + brief(put(longer, "image/png", png)));
|
||||
|
||||
// download with a Content-Disposition override
|
||||
HttpResponse<String> dl = Cap.HTTP.send(HttpRequest.newBuilder(URI.create(
|
||||
"http://localhost:" + port + "/presign/download/" + key + "?filename=holiday.png")).build(), HttpResponse.BodyHandlers.ofString());
|
||||
String getUrl = json.readTree(dl.body()).get("url").asString();
|
||||
HttpResponse<byte[]> got = Cap.HTTP.send(HttpRequest.newBuilder(URI.create(getUrl)).build(), HttpResponse.BodyHandlers.ofByteArray());
|
||||
Cap.say(F, "GET presigned URL -> status=" + got.statusCode() + " bytes match upload=" + Arrays.equals(got.body(), png)
|
||||
+ " content-disposition=" + got.headers().firstValue("content-disposition").orElse("(absent)"));
|
||||
|
||||
Cap.say(F, "disallowed type: POST /presign/upload text/html -> status=" + Cap.HTTP.send(HttpRequest.newBuilder(
|
||||
URI.create("http://localhost:" + port + "/presign/upload")).header("Content-Type", "application/json")
|
||||
.POST(HttpRequest.BodyPublishers.ofString("{\"contentType\":\"text/html\"}")).build(), HttpResponse.BodyHandlers.ofString()).statusCode());
|
||||
}
|
||||
|
||||
@Test
|
||||
void expiredUrlIsRejected() throws Exception {
|
||||
S3PresignService.Upload u = presign.forUpload("image/png", Duration.ofSeconds(1));
|
||||
Thread.sleep(2200);
|
||||
HttpResponse<String> r = put(u.url(), "image/png", "x".getBytes(StandardCharsets.UTF_8));
|
||||
Cap.say(F, "PUT a URL signed for 1 second, sent after 2.2 seconds -> " + brief(r));
|
||||
assertThat(r.statusCode()).isEqualTo(403);
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user