Add graphrag-neo4j module: structured-output entity and relation extraction into a typed Neo4j graph with a vector index, multi-hop retrieval by span linking and variable-length Cypher paths, compared with plain vector RAG over the same 35 chunks and 9 questions (complete evidence 7/9 vs 5/9 at k=8); Testcontainers on neo4j:2026.10.1 and 5.26. The extraction step is a rule-based stand-in, not a language model.
Co-Authored-By: Claude Sonnet 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01CifR2BE7PBvRy5hT8TS9oz
This commit is contained in:
Claude Sonnet 5.5
@@ -0,0 +1,76 @@
|
||||
# Entity and relation extraction through ChatClient.entity() (STUB model, not an LLM)
|
||||
|
||||
|
||||
--- the user message the model receives for svc-checkout.txt ---
|
||||
Extract the entities and relations from the document.
|
||||
|
||||
DOCUMENT_START
|
||||
Checkout Service. The Checkout Service takes a customer's cart and turns it into an order. It depends on the Payments Gateway to take card payments and on the Inventory Service to reserve stock. The Checkout Service is owned by the Storefront Team. Its latency dashboard lives in the shared Grafana folder.
|
||||
DOCUMENT_END
|
||||
|
||||
Your response should be in JSON format.
|
||||
Do not include any explanations, only provide a RFC8259 compliant JSON response following this format without deviation.
|
||||
Do not include markdown code blocks in your response.
|
||||
Remove the ```json markdown from the output.
|
||||
Here is the JSON Schema instance your output must adhere to:
|
||||
```{
|
||||
"$schema" : "https://json-schema.org/draft/2020-12/schema",
|
||||
"type" : "object",
|
||||
"properties" : {
|
||||
"entities" : {
|
||||
"type" : "array",
|
||||
"items" : {
|
||||
"type" : "object",
|
||||
"properties" : {
|
||||
"name" : {
|
||||
"type" : "string"
|
||||
},
|
||||
"type" : {
|
||||
"type" : "string",
|
||||
"enum" : [ "SYSTEM", "TEAM", "PERSON", "VENDOR", "INCIDENT", "PLACE" ]
|
||||
}
|
||||
},
|
||||
"required" : [ "name", "type" ],
|
||||
"additionalProperties" : false
|
||||
}
|
||||
},
|
||||
"relations" : {
|
||||
"type" : "array",
|
||||
"items" : {
|
||||
"type" : "object",
|
||||
"properties" : {
|
||||
"source" : {
|
||||
"type" : "string"
|
||||
},
|
||||
"target" : {
|
||||
"type" : "string"
|
||||
},
|
||||
"type" : {
|
||||
"type" : "string",
|
||||
"enum" : [ "DEPENDS_ON", "OWNED_BY", "LED_BY", "HOSTED_ON", "AFFECTED", "CAUSED_BY", "REPORTS_TO", "LOCATED_IN" ]
|
||||
}
|
||||
},
|
||||
"required" : [ "source", "target", "type" ],
|
||||
"additionalProperties" : false
|
||||
}
|
||||
}
|
||||
},
|
||||
"required" : [ "entities", "relations" ],
|
||||
"additionalProperties" : false
|
||||
}```
|
||||
|
||||
--- the parsed Extraction record ---
|
||||
entity SYSTEM Checkout Service
|
||||
entity SYSTEM Payments Gateway
|
||||
entity SYSTEM Inventory Service
|
||||
entity TEAM Storefront Team
|
||||
relation Checkout Service -[DEPENDS_ON]-> Payments Gateway
|
||||
relation Checkout Service -[DEPENDS_ON]-> Inventory Service
|
||||
relation Checkout Service -[OWNED_BY]-> Storefront Team
|
||||
|
||||
--- all 35 chunks ---
|
||||
model calls : 35 (one per chunk)
|
||||
characters sent : 72437 (about 18109 tokens at 4 characters per token, an estimate)
|
||||
entity mentions (before merging): 54
|
||||
relations : 33
|
||||
chunks with no relation at all : 14
|
||||
@@ -0,0 +1,20 @@
|
||||
# What a bad model reply does to extraction (replies produced by the STUB's failure modes)
|
||||
|
||||
|
||||
--- JSON wrapped in a markdown fence (reply starts with the fence) ---
|
||||
parsed OK, 2 relations
|
||||
|
||||
--- the same fence, after one sentence of chat ---
|
||||
tools.jackson.core.exc.StreamReadException: Unrecognized token 'Here': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
|
||||
|
||||
--- JSON cut off two thirds of the way through ---
|
||||
tools.jackson.core.exc.UnexpectedEndOfInputException: Unexpected end-of-input in property name
|
||||
|
||||
--- a relation type outside the allowed list (IS_RESPONSIBLE_FOR) ---
|
||||
tools.jackson.databind.exc.InvalidFormatException: Cannot deserialize value of type `com.ankurm.graphrag.RelationType` from String "IS_RESPONSIBLE_FOR": not one of the values accepted for Enum class: [OWNED_BY, LOCATED_IN, AFFECTED, HOSTED_ON, LED_BY, DEPENDS_ON, REPORTS...
|
||||
|
||||
--- a relation type that is a Cypher fragment ---
|
||||
tools.jackson.databind.exc.InvalidFormatException: Cannot deserialize value of type `com.ankurm.graphrag.RelationType` from String "OWNED_BY]->(x) DETACH DELETE x //": not one of the values accepted for Enum class: [OWNED_BY, LOCATED_IN, AFFECTED, HOSTED_ON, LED_BY, DEPE...
|
||||
|
||||
--- a relation to an entity that was never listed ---
|
||||
relations kept 2, dropped 1
|
||||
@@ -0,0 +1,45 @@
|
||||
# Building the graph in Neo4j; names that are the same thing (STUB extraction)
|
||||
|
||||
image : neo4j:2026.10.1-community
|
||||
server : Neo4j Kernel 2026.10.1 community
|
||||
|
||||
--- vector indexes (one from Neo4jVectorStore for chunks, one of ours for entity names) ---
|
||||
chunk_embedding on (:Chunk).embedding ONLINE
|
||||
entity_name_embedding on (:Entity).embedding ONLINE
|
||||
|
||||
--- A. exact-name matching only (no alias table, similarity merge off) ---
|
||||
entities: 28
|
||||
Payments Gateway : -[DEPENDS_ON]-> Ledger Service; -[HOSTED_ON]-> Arcadia Cloud; -[OWNED_BY]-> Payments Team; <-[DEPENDS_ON]- Checkout Service
|
||||
Card Gateway : <-[AFFECTED]- INC-2060
|
||||
|
||||
--- same-type name pairs by cosine of their embeddings, top 8 of 72 ---
|
||||
0.806 INCIDENT INC-2057 <-> INC-2060
|
||||
0.797 INCIDENT INC-2041 <-> INC-2057
|
||||
0.758 INCIDENT INC-2041 <-> INC-2060
|
||||
0.681 SYSTEM Card Gateway <-> Payments Gateway
|
||||
0.540 SYSTEM Ledger Service <-> Inventory Service
|
||||
0.535 TEAM Storefront Team <-> Platform Team
|
||||
0.530 TEAM Core Data Team <-> Platform Team
|
||||
0.519 SYSTEM Inventory Service <-> Checkout Service
|
||||
-- Card Gateway <-> Payments Gateway: cosine 0.681, rank 4
|
||||
|
||||
--- what a similarity-only merge would do at each threshold ---
|
||||
threshold 0.90 -> 0 merges:
|
||||
threshold 0.80 -> 1 merges: INC-2057 + INC-2060
|
||||
threshold 0.70 -> 3 merges: INC-2057 + INC-2060; INC-2041 + INC-2057; INC-2041 + INC-2060
|
||||
threshold 0.60 -> 4 merges: INC-2057 + INC-2060; INC-2041 + INC-2057; INC-2041 + INC-2060; Card Gateway + Payments Gateway
|
||||
|
||||
--- B. plus a one-line alias table (Card Gateway = Payments Gateway) ---
|
||||
entities: 27
|
||||
Payments Gateway : -[DEPENDS_ON]-> Ledger Service; -[HOSTED_ON]-> Arcadia Cloud; -[OWNED_BY]-> Payments Team; <-[AFFECTED]- INC-2060; <-[DEPENDS_ON]- Checkout Service
|
||||
chunks: 35, MENTIONS edges: 54, dropped relations: 0
|
||||
|
||||
--- relationships by type ---
|
||||
DEPENDS_ON 6
|
||||
OWNED_BY 6
|
||||
HOSTED_ON 5
|
||||
LED_BY 5
|
||||
REPORTS_TO 4
|
||||
AFFECTED 3
|
||||
CAUSED_BY 2
|
||||
LOCATED_IN 2
|
||||
@@ -0,0 +1,45 @@
|
||||
# Building the graph in Neo4j; names that are the same thing (STUB extraction)
|
||||
|
||||
image : neo4j:5-community
|
||||
server : Neo4j Kernel 5.26.32 community
|
||||
|
||||
--- vector indexes (one from Neo4jVectorStore for chunks, one of ours for entity names) ---
|
||||
chunk_embedding on (:Chunk).embedding ONLINE
|
||||
entity_name_embedding on (:Entity).embedding ONLINE
|
||||
|
||||
--- A. exact-name matching only (no alias table, similarity merge off) ---
|
||||
entities: 28
|
||||
Payments Gateway : -[DEPENDS_ON]-> Ledger Service; -[HOSTED_ON]-> Arcadia Cloud; -[OWNED_BY]-> Payments Team; <-[DEPENDS_ON]- Checkout Service
|
||||
Card Gateway : <-[AFFECTED]- INC-2060
|
||||
|
||||
--- same-type name pairs by cosine of their embeddings, top 8 of 72 ---
|
||||
0.806 INCIDENT INC-2057 <-> INC-2060
|
||||
0.797 INCIDENT INC-2041 <-> INC-2057
|
||||
0.758 INCIDENT INC-2041 <-> INC-2060
|
||||
0.681 SYSTEM Card Gateway <-> Payments Gateway
|
||||
0.540 SYSTEM Ledger Service <-> Inventory Service
|
||||
0.535 TEAM Storefront Team <-> Platform Team
|
||||
0.530 TEAM Core Data Team <-> Platform Team
|
||||
0.519 SYSTEM Inventory Service <-> Checkout Service
|
||||
-- Card Gateway <-> Payments Gateway: cosine 0.681, rank 4
|
||||
|
||||
--- what a similarity-only merge would do at each threshold ---
|
||||
threshold 0.90 -> 0 merges:
|
||||
threshold 0.80 -> 1 merges: INC-2057 + INC-2060
|
||||
threshold 0.70 -> 3 merges: INC-2057 + INC-2060; INC-2041 + INC-2057; INC-2041 + INC-2060
|
||||
threshold 0.60 -> 4 merges: INC-2057 + INC-2060; INC-2041 + INC-2057; INC-2041 + INC-2060; Card Gateway + Payments Gateway
|
||||
|
||||
--- B. plus a one-line alias table (Card Gateway = Payments Gateway) ---
|
||||
entities: 27
|
||||
Payments Gateway : -[DEPENDS_ON]-> Ledger Service; -[HOSTED_ON]-> Arcadia Cloud; -[OWNED_BY]-> Payments Team; <-[AFFECTED]- INC-2060; <-[DEPENDS_ON]- Checkout Service
|
||||
chunks: 35, MENTIONS edges: 54, dropped relations: 0
|
||||
|
||||
--- relationships by type ---
|
||||
DEPENDS_ON 6
|
||||
OWNED_BY 6
|
||||
HOSTED_ON 5
|
||||
LED_BY 5
|
||||
REPORTS_TO 4
|
||||
AFFECTED 3
|
||||
CAUSED_BY 2
|
||||
LOCATED_IN 2
|
||||
@@ -0,0 +1,31 @@
|
||||
# Plain vector RAG against GraphRAG: is every supporting document in the context?
|
||||
|
||||
corpus: 35 chunks (one document each); 9 questions; evidence = the documents that together hold every fact needed
|
||||
vector: Neo4jVectorStore, top-k chunks. graph: link spans (cosine >= 0.80), walk <= 3 hops, keep the 6 best-ranked paths, take the chunks that stated their edges
|
||||
|
||||
id hops | vector k4 vector k8 | graph chunks | question
|
||||
Q1 1 | yes yes | yes 6 | Who leads the Fulfilment Team?
|
||||
Q2 2 | yes yes | yes 5 | Who leads the team that owns the Payments Gateway?
|
||||
Q3 2 | NO NO | yes 4 | Which team owns the root cause of INC-2041?
|
||||
Q4 3 | NO yes | yes 5 | Who does the leader of the team that owns the Payments Gateway report to?
|
||||
Q5 3 | NO yes | yes 4 | Where is the company that hosts the database behind the Ledger Service located?
|
||||
Q6 3 | NO yes | NO 5 | Who leads the teams that own the services the Checkout Service depends on?
|
||||
Q7 3 | NO NO | NO 5 | Which services stop working if the Orders Database goes down?
|
||||
Q8 2 | NO NO | yes 6 | Which incidents affected services that the Checkout Service depends on directly?
|
||||
Q9 3 | NO NO | yes 4 | Who leads the team that owns the service INC-2057 affected?
|
||||
|
||||
complete evidence: vector k=4 2/9, vector k=8 5/9, graph 7/9 (graph context averaged 4.9 chunks; vector k=8 is always 8)
|
||||
1-hop questions (1): vector k=4 1, vector k=8 1, graph 1
|
||||
2-hop questions (3): vector k=4 1, vector k=8 1, graph 3
|
||||
3-hop questions (5): vector k=4 0, vector k=8 3, graph 3
|
||||
|
||||
--- what each retriever missed ---
|
||||
Q1 vector k=8 missing [] graph missing []
|
||||
Q2 vector k=8 missing [] graph missing []
|
||||
Q3 vector k=8 missing [db-orders] graph missing []
|
||||
Q4 vector k=8 missing [] graph missing []
|
||||
Q5 vector k=8 missing [] graph missing []
|
||||
Q6 vector k=8 missing [] graph missing [team-payments]
|
||||
Q7 vector k=8 missing [svc-payments-gateway] graph missing [svc-checkout, svc-payments-gateway]
|
||||
Q8 vector k=8 missing [inc-2057] graph missing []
|
||||
Q9 vector k=8 missing [team-fulfilment, svc-inventory] graph missing []
|
||||
@@ -0,0 +1,32 @@
|
||||
# How the graph retriever's own settings move the result (complete evidence out of 9)
|
||||
|
||||
|
||||
--- hops walked x paths kept (link threshold 0.80) ---
|
||||
max hops | 6 paths 10 paths 20 paths 40 paths
|
||||
1 | 3/9 ( 3.0 ch) 3/9 ( 3.4 ch) 3/9 ( 3.4 ch) 3/9 ( 3.4 ch)
|
||||
2 | 4/9 ( 3.9 ch) 4/9 ( 5.1 ch) 7/9 ( 7.8 ch) 7/9 ( 8.4 ch)
|
||||
3 | 7/9 ( 4.9 ch) 8/9 ( 6.8 ch) 9/9 (10.3 ch) 9/9 (13.6 ch)
|
||||
|
||||
--- entity-link threshold (3 hops, 6 paths) ---
|
||||
threshold | complete seeds per question (avg)
|
||||
0.70 | 7/9 3.9
|
||||
0.80 | 7/9 1.9
|
||||
0.85 | 8/9 1.2
|
||||
0.90 | 8/9 1.0
|
||||
0.95 | 8/9 1.0
|
||||
|
||||
--- the seeds found for three questions at 0.80 (span -> entity, cosine) ---
|
||||
Q2 Who leads the team that owns the Payments Gateway?
|
||||
'Payments Gateway' -> Payments Gateway (1.00)
|
||||
'Payments' -> Payments Team (0.88)
|
||||
'team' -> Fulfilment Team (0.83)
|
||||
Q5 Where is the company that hosts the database behind the Ledger Service located?
|
||||
'Ledger Service' -> Ledger Service (1.00)
|
||||
'database' -> Orders Database (0.82)
|
||||
Q7 Which services stop working if the Orders Database goes down?
|
||||
'Orders Database' -> Orders Database (1.00)
|
||||
|
||||
--- candidate paths from one seed as the hop limit grows ---
|
||||
Payments Gateway 1 hop : 5 2 hops: 14 3 hops: 32 4 hops: 54
|
||||
Orders Database 1 hop : 4 2 hops: 12 3 hops: 24 4 hops: 46
|
||||
Checkout Service 1 hop : 3 2 hops: 12 3 hops: 27 4 hops: 49
|
||||
@@ -0,0 +1,20 @@
|
||||
# The prompt GraphRAG assembles for Q4 (no model was called; nothing below is a model answer)
|
||||
|
||||
QUESTION: Who does the leader of the team that owns the Payments Gateway report to?
|
||||
|
||||
FACTS FROM THE GRAPH:
|
||||
- Payments Gateway is owned by Payments Team
|
||||
- Payments Gateway is owned by Payments Team; Payments Team is led by Priya Raman; Priya Raman reports to Dana Okafor
|
||||
- Payments Gateway is owned by Payments Team; Payments Team is led by Priya Raman
|
||||
- Payments Gateway is owned by Payments Team; INC-2060 affected Payments Gateway
|
||||
- Payments Gateway is owned by Payments Team; Checkout Service depends on Payments Gateway
|
||||
- Payments Gateway is owned by Payments Team; Payments Gateway is hosted on Arcadia Cloud
|
||||
|
||||
SOURCE PASSAGES:
|
||||
[svc-payments-gateway] Payments Gateway. The Payments Gateway talks to the card networks on behalf of every other service. The Payments Gateway depends on the Ledger Service to record each authorised charge. The Payments Gateway is owned by the Payments Team. It is hosted on Arcadia Cloud.
|
||||
[team-payments] Payments Team. The Payments Team looks after card acceptance and refunds. The Payments Team is led by Priya Raman. The Payments Team is based in Pune. The team holds a weekly review on Thursdays.
|
||||
[person-priya] Priya Raman. Priya Raman joined the company in 2021 as a payments engineer. Priya Raman reports to Dana Okafor. She is the named contact for card network certificate renewals.
|
||||
[inc-2060] INC-2060. INC-2060 was a 12-minute slowdown on card payments. INC-2060 affected the Card Gateway. The Card Gateway is the on-call channel's name for the Payments Gateway. A retry storm after a network blip was the trigger. The write-up recommends capping retries.
|
||||
[svc-checkout] Checkout Service. The Checkout Service takes a customer's cart and turns it into an order. It depends on the Payments Gateway to take card payments and on the Inventory Service to reserve stock. The Checkout Service is owned by the Storefront Team. Its latency dashboard lives in the shared Grafana folder.
|
||||
|
||||
characters in this prompt: 1922; characters in the 8 chunks a k=8 vector search would add instead: 1820
|
||||
@@ -0,0 +1,23 @@
|
||||
# Questions whose shape is known: the Cypher is the retrieval (hand-written, parameterised, run on the real graph)
|
||||
|
||||
|
||||
--- Q7 Which services stop working if the Orders Database goes down? ---
|
||||
MATCH (db:Entity {name: $name})<-[:DEPENDS_ON*1..5]-(s:Entity)
|
||||
RETURN DISTINCT s.name AS service ORDER BY service
|
||||
-> Checkout Service
|
||||
-> Ledger Service
|
||||
-> Payments Gateway
|
||||
|
||||
--- Q6 Who leads the teams that own the services the Checkout Service depends on? ---
|
||||
MATCH (c:Entity {name: $name})-[:DEPENDS_ON]->(d:Entity)-[:OWNED_BY]->(team:Entity)-[:LED_BY]->(lead:Entity)
|
||||
RETURN d.name AS dependency, team.name AS team, lead.name AS lead ORDER BY dependency
|
||||
-> Inventory Service / Fulfilment Team / Tomasz Nowak
|
||||
-> Payments Gateway / Payments Team / Priya Raman
|
||||
|
||||
--- Q8 entity linking and the traversal in ONE statement: the vector index finds the seed, MATCH walks from it ---
|
||||
CALL db.index.vector.queryNodes('entity_name_embedding', 1, $vector) YIELD node AS c, score
|
||||
MATCH (c)-[:DEPENDS_ON]->(d:Entity)<-[:AFFECTED]-(i:Entity)
|
||||
RETURN c.name AS seed, score, d.name AS dependency, i.name AS incident ORDER BY incident
|
||||
$vector = embedding of the span "Checkout Service"
|
||||
-> seed Checkout Service (1.00): INC-2057 affected Inventory Service
|
||||
-> seed Checkout Service (1.00): INC-2060 affected Payments Gateway
|
||||
Reference in New Issue
Block a user