asmhatreandClaude Sonnet 5.5
108fc44542
Co-Authored-By: Claude Sonnet 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01CifR2BE7PBvRy5hT8TS9oz
32 lines
2.3 KiB
Plaintext
32 lines
2.3 KiB
Plaintext
# Plain vector RAG against GraphRAG: is every supporting document in the context?
|
|
|
|
corpus: 35 chunks (one document each); 9 questions; evidence = the documents that together hold every fact needed
|
|
vector: Neo4jVectorStore, top-k chunks. graph: link spans (cosine >= 0.80), walk <= 3 hops, keep the 6 best-ranked paths, take the chunks that stated their edges
|
|
|
|
id hops | vector k4 vector k8 | graph chunks | question
|
|
Q1 1 | yes yes | yes 6 | Who leads the Fulfilment Team?
|
|
Q2 2 | yes yes | yes 5 | Who leads the team that owns the Payments Gateway?
|
|
Q3 2 | NO NO | yes 4 | Which team owns the root cause of INC-2041?
|
|
Q4 3 | NO yes | yes 5 | Who does the leader of the team that owns the Payments Gateway report to?
|
|
Q5 3 | NO yes | yes 4 | Where is the company that hosts the database behind the Ledger Service located?
|
|
Q6 3 | NO yes | NO 5 | Who leads the teams that own the services the Checkout Service depends on?
|
|
Q7 3 | NO NO | NO 5 | Which services stop working if the Orders Database goes down?
|
|
Q8 2 | NO NO | yes 6 | Which incidents affected services that the Checkout Service depends on directly?
|
|
Q9 3 | NO NO | yes 4 | Who leads the team that owns the service INC-2057 affected?
|
|
|
|
complete evidence: vector k=4 2/9, vector k=8 5/9, graph 7/9 (graph context averaged 4.9 chunks; vector k=8 is always 8)
|
|
1-hop questions (1): vector k=4 1, vector k=8 1, graph 1
|
|
2-hop questions (3): vector k=4 1, vector k=8 1, graph 3
|
|
3-hop questions (5): vector k=4 0, vector k=8 3, graph 3
|
|
|
|
--- what each retriever missed ---
|
|
Q1 vector k=8 missing [] graph missing []
|
|
Q2 vector k=8 missing [] graph missing []
|
|
Q3 vector k=8 missing [db-orders] graph missing []
|
|
Q4 vector k=8 missing [] graph missing []
|
|
Q5 vector k=8 missing [] graph missing []
|
|
Q6 vector k=8 missing [] graph missing [team-payments]
|
|
Q7 vector k=8 missing [svc-payments-gateway] graph missing [svc-checkout, svc-payments-gateway]
|
|
Q8 vector k=8 missing [inc-2057] graph missing []
|
|
Q9 vector k=8 missing [team-fulfilment, svc-inventory] graph missing []
|