spring-grpc-boot4/docs/01-troubleshooting.md

Troubleshooting Spring gRPC on Boot 4

Symptom-first. Each entry is something whose error message points somewhere other than its cause. Entries marked [tested] are reproduced in HardToDiagnoseTest; entries marked [documented] need a real network and are stated rather than asserted, because a test claiming to prove them over in-process transport would be lying.

---

UNAVAILABLE: Unable to resolve host <name> — [tested]

Cause, 90% of the time: a typo in a property name, not a networking problem.

An unmatched channel name is passed straight to the DNS resolver as a target. Boot does not warn that the channel is unconfigured. The exact names, both of which are easy to get wrong:

spring.grpc.server.inprocess.name=orders-test          # NOT "in-process"
spring.grpc.client.channel.orders.target=static://...   # SINGULAR "channel"; "target", NOT "address"

Also check spring.grpc.client.inprocess.enabled=true if you are using the in-process transport from the client side.

IllegalStateException: No grpc channel factory found that supports target : <name>

GrpcChannelFactory is a composite that asks each registered factory supports(target) before named-channel indirection is applied. Passing a bare logical name to createChannel() can therefore fail even though the name is correctly configured. In application code, inject the stub (via @ImportGrpcClients) and let Boot resolve the target; in tests, pass the full target.

RESOURCE_EXHAUSTED: gRPC message exceeds maximum size 4194304 — [tested]

The default limit is 4 MB, per message, on the receiver, and the two sides are configured separately.

# large RESPONSE -> the CLIENT is the receiver
spring.grpc.client.channel.<name>.inbound.message.max-size=16MB
# large REQUEST  -> the SERVER is the receiver
spring.grpc.server.inbound.message.max-size=16MB

Per call: stub.withMaxInboundMessageSize(16 * 1024 * 1024).

Half of all "I raised the limit and it still fails" reports are the asymmetry: raising it on the server does nothing for a large response.

The testing trap. The in-process transport passes messages by reference and never serialises them, so it cannot enforce size limits. A payload-size regression passes every in-process test you have and fails in production. The same applies to compression and anything else depending on the wire format. This is demonstrated — the test asserts that a 4 MB + 1 KB message goes through in-process without complaint.

Prefer streaming to large unary messages: the limit is per message, so 1,000 small messages are fine. Do not raise the limit globally — a large limit turns a malformed request into an OOM.

Calls that hang forever — [tested]

gRPC has no default deadline. A call without one waits indefinitely. Nothing warns you, and in testing the server always responds quickly, so it surfaces the day something downstream is slow — as a thread pool that fills up with no errors in the log.

stub.withDeadlineAfter(2, TimeUnit.SECONDS).getOrder(request);

Per channel: spring.grpc.client.channel.<name>.default.deadline=2s, or register a DefaultDeadlineSetupClientInterceptor.

Deadlines are absolute and propagate: if A calls B with 2s remaining, B sees 2s, not a fresh 2s. Set the deadline at the edge; do not re-set it at every hop, or a deep chain multiplies its budget.

The server keeps working after the client gave up — [tested]

gRPC does not interrupt your thread on cancellation. It sets a flag on the Context.

for (Item item : items) {
    if (Context.current().isCancelled()) return;   // no onCompleted/onError -- stream is closed
    responseObserver.onNext(convert(item));
}

For blocking work handed to another thread, propagate with Context.current().wrap(runnable) — otherwise the flag is invisible there. Calling onNext/onCompleted after cancellation throws IllegalStateException.

Every error is UNKNOWN with no message — [tested]

Any exception escaping a handler becomes UNKNOWN with a null description — deliberately, since leaking exception text across a service boundary is an information-disclosure risk.

Metadata trailers = new Metadata();
trailers.put(REASON_KEY, "ORDER_LOCKED");
responseObserver.onError(new StatusRuntimeException(
        Status.FAILED_PRECONDITION.withDescription("order is locked"), trailers));

Choose the code carefully — it is the contract that retry policies, circuit breakers and dashboards key off:

Do not retry	Retry may help
INVALID_ARGUMENT, NOT_FOUND, ALREADY_EXISTS, FAILED_PRECONDITION, PERMISSION_DENIED, UNAUTHENTICATED	UNAVAILABLE, DEADLINE_EXCEEDED, RESOURCE_EXHAUSTED, ABORTED

Getting this wrong makes non-retryable failures look retryable and turns one bad request into a storm.

IllegalStateException: call already closed / half-closed

You called onNext or onCompleted after the stream ended. Usual causes:

• responding after cancellation or deadline expiry (check Context.current().isCancelled() first);
• calling onCompleted() twice;
• calling onError() and then onCompleted() — onError is terminal;
• two threads writing to one StreamObserver. StreamObserver is not thread-safe. Concurrent onNext corrupts the stream, and the symptom is usually a deserialization error on the far side, pointing nowhere near the bug.

A client-streaming call never completes

You forgot the half-close. requestObserver.onCompleted() is what tells the server "no more requests"; without it the server's onCompleted never fires and the call hangs until the deadline — or forever, if you did not set one.

Retries do nothing — [documented]

gRPC retries are configured through the service config, not a client API, and are off by default:

spring.grpc.client.channel.orders.service-config.methodConfig[0].retryPolicy.maxAttempts=4
spring.grpc.client.channel.orders.service-config.methodConfig[0].retryPolicy.initialBackoff=0.5s
spring.grpc.client.channel.orders.service-config.methodConfig[0].retryPolicy.retryableStatusCodes[0]=UNAVAILABLE

Three reasons they silently do nothing:

1. The status you are failing with is not in retryableStatusCodes.
2. The response was already partially delivered — gRPC will not retry a committed stream.
3. Retry is disabled on the channel (enableRetry() / the corresponding property).

Only make idempotent methods retryable. gRPC has no idea whether your RPC is safe to repeat.

Connections drop every N minutes through a proxy — [documented]

Load balancers and ingress controllers close idle HTTP/2 connections. gRPC reuses one long-lived connection, so an idle stream is not an idle connection — the proxy disagrees.

spring.grpc.client.channel.<name>.keepalive.time=30s
spring.grpc.client.channel.<name>.keepalive.timeout=5s
spring.grpc.server.keepalive.connection.max-idle-time=...
spring.grpc.server.keepalive.connection.max-age=...

Keep the client's keepalive interval shorter than the proxy's idle timeout. Note the server can reject keepalives that are too frequent (ENHANCE_YOUR_CALM / too_many_pings), which looks like a random disconnect — so both ends need to agree.

All traffic goes to one backend — [documented]

The default load-balancing policy is pick_first, not round_robin. With a DNS target resolving to several addresses, a gRPC client picks one connection and keeps it. This is correct behaviour and almost never what people expect behind a headless Kubernetes service.

Set round_robin via service config, and remember that DNS re-resolution happens on connection failure, not on a timer — so scaling up does not redistribute existing connections.

---

Test transport limitations, collected

The in-process transport is excellent for testing protocol behaviour and misleading for anything wire-related. It does exercise interceptors, metadata, status codes, deadlines, cancellation and flow control. It does not exercise:

• message size limits (no serialization — proven in the test suite),
• compression,
• TLS and credentials negotiation,
• keepalive, GOAWAY, idle timeouts,
• name resolution and load balancing,
• HTTP/2 framing and header limits.

Anything in that second list needs a real port, and belongs in a smaller, slower test tier.