spring-ai/rag/docs/06-observability-and-production-checklist.md

6. Observability and a production checklist

prev: 5. Generation and the faithfulness check · Index

What the application exports

GET /actuator/prometheus from the end-to-end run (output 10) contained these rag_ series after two ingested files, one skipped re-upload and two questions:

rag_chunks_ingested_total 5.0
rag_context_chunks_count 2
rag_context_chunks_sum 5.0
rag_context_chunks_max 4.0
rag_ingestion_skipped_total 1.0
rag_queries_total{status="answered"} 2.0
rag_query_duration_seconds_count{status="answered"} 2

Counters that exist in the code but were not triggered in that run (so they do not appear): rag_rerank_failures_total and rag_faithfulness_judge_failures_total. Micrometer registers a counter when it is first incremented.

The ones worth alerting on: a rising share of ungrounded and no_context in rag_queries_total; any increase in rag_rerank_failures_total (silent degradation, chapter 4); any increase in rag_faithfulness_judge_failures_total; and rag_context_chunks drifting toward its maximum (the threshold is accepting everything).

Checklist

• Threshold set from questions your corpus cannot answer (chapter 4), not copied from a tutorial.
• allowEmptyContext(false), and a test that an off-topic question produces no_context.
• Tenant filter built with FilterExpressionBuilder, never string concatenation; a test with a hostile tenant id.
• Filtered queries tested on a table the size of production (HNSW and WHERE, chapter 4).
• The ingestion tracker persisted, or deletion by source_file filter (chapter 3).
• Judge prompt forces a one-word answer; ungrounded is shown to the user as such, not hidden.
• Reranker replies parsed with a fallback and the failure counter alerted (chapter 4). Consider a structured-output call instead of parsing text.
• init.sql (or your migration tool) owns the schema; initialize-schema stays false.
• Embedding model and vector dimension changed together, and the table rebuilt when either changes: vectors from two models are not comparable.
• An evaluation set of real questions with known answers, run on every change to chunking, threshold or models. This repository has none; it is the largest gap between this code and a production system.