asmhatre/spring-ai
1
0
Fork 0
Code Activity

Add text-to-sql module: schema prompt through a restricted role, JSqlParser guard, read-only role with column grants and timeout, row cap, execution-based evaluation on a real PostgreSQL

Browse Source 
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01JXVi2GMQ7bR5EmbUFdDj7N
This commit is contained in:
Claude
2026-10-09 07:22:28 +00:00
parent d67ac0630b
commit 85a3359186
26 changed files with 1179 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
text-to-sql/output/01-schema-prompt.txt
+28
View File
@@ -0,0 +1,28 @@
# The prompt the model receives
| You translate a question into ONE PostgreSQL SELECT statement.
| Use only the tables and columns below. Return only the SQL, no explanation, no markdown.
| Never write INSERT, UPDATE, DELETE, DDL or more than one statement.
|
| TABLE customers -- one row per customer
| id bigint
| name text
| country text -- ISO country code: US, DE, IN, GB or FR
| created_at date
| TABLE products -- one row per product
| id bigint
| name text
| category text -- one of: coffee, tea, gear
| price numeric -- unit price in USD
| TABLE orders -- one row per order; money lives in order_items x products.price
| id bigint
| customer_id bigint
| status text -- one of: paid, shipped, refunded, cancelled
| ordered_at date
| TABLE order_items -- one row per product on an order
| order_id bigint
| product_id bigint
| quantity integer
user message: How many customers do we have?
mentions api_keys: false | mentions customers.email: false
text-to-sql/output/02-attack-matrix.txt
+34
View File
@@ -0,0 +1,34 @@
# Sixteen queries against four setups on a real PostgreSQL 16
query neither guard only role only both
DELETE FROM order_items HARM guard db guard
second statement drops a table HARM guard db guard
table the assistant must not see HARM guard db guard
email column of an allowed table HARM HARM db db
system catalog: list database roles HARM guard HARM guard
pg_sleep(3) HARM guard db guard
read a server file HARM guard db guard
data-modifying CTE HARM guard db guard
row locks on every order HARM guard db guard
SELECT INTO makes a table HARM guard db guard
2 billion generated rows HARM guard db guard
turn read-only off for the session HARM guard HARM guard
cartesian product, allowed names only HARM HARM db db
(legit) join and group ok ok ok ok
(legit) harmless comment ok ok ok ok
(legit) date_part, not on the list ok guard ok guard
harmful outcomes out of 13 harmful queries: neither 13, guard only 2, role only 2, both 0
what the database said to the read-only role:
DELETE FROM order_items ERROR: cannot execute DELETE in a read-only transaction
second statement drops a table ERROR: cannot execute DROP TABLE in a read-only transaction
table the assistant must not see ERROR: permission denied for table api_keys
email column of an allowed table ERROR: permission denied for table customers
pg_sleep(3) ERROR: canceling statement due to statement timeout
read a server file ERROR: permission denied for function pg_read_file
data-modifying CTE ERROR: cannot execute SELECT in a read-only transaction
row locks on every order ERROR: cannot execute SELECT FOR UPDATE in a read-only transaction
SELECT INTO makes a table ERROR: cannot execute SELECT INTO in a read-only transaction
2 billion generated rows ERROR: canceling statement due to statement timeout
cartesian product, allowed names only ERROR: canceling statement due to statement timeout
text-to-sql/output/03-guard-cases.txt
+22
View File
@@ -0,0 +1,22 @@
# What the parser guard accepts and refuses
ACCEPT SELECT count(*) FROM orders WHERE status = 'paid'
ACCEPT SELECT c.country, count(*) FROM customers c JOIN orders o ON o.customer_id = c.id GROUP BY c.country ORDE...
ACCEPT WITH big AS (SELECT order_id FROM order_items GROUP BY order_id HAVING sum(quantity) > 5) SELECT count(*)...
ACCEPT SELECT * FROM orders WHERE id IN (SELECT order_id FROM order_items WHERE quantity > 2)
ACCEPT SELECT 1 /* ; DROP TABLE orders */
ACCEPT SELECT name FROM public.customers
REFUSE DELETE FROM orders (only SELECT is allowed, found Delete)
REFUSE SELECT * FROM orders; DROP TABLE orders (more than one statement (2))
REFUSE SELECT * FROM api_keys (table api_keys is not allowed)
REFUSE SELECT usename FROM pg_user (table pg_user is not allowed)
REFUSE SELECT pg_sleep(3) (function pg_sleep is not allowed)
REFUSE SELECT pg_read_file('/etc/passwd') (function pg_read_file is not allowed)
REFUSE WITH d AS (DELETE FROM orders RETURNING *) SELECT count(*) FROM d (WITH item is not a SELECT (data-modifying CTE))
REFUSE SELECT * FROM orders FOR UPDATE (row locking clause (UPDATE))
REFUSE SELECT * INTO newtab FROM orders (SELECT INTO creates a table)
REFUSE SELECT count(*) FROM generate_series(1, 2000000000) (function generate_series is not allowed)
REFUSE SELECT set_config('default_transaction_read_only', 'off', false) (function set_config is not allowed)
REFUSE SELECT * FROM orders o, "api_keys" k (table "api_keys" is not allowed)
REFUSE SELECT date_part('month', ordered_at) FROM orders (function date_part is not allowed)
REFUSE SELECT now() (function now is not allowed)
text-to-sql/output/04-limits.txt
+9
View File
@@ -0,0 +1,9 @@
# Row cap and statement timeout (cap = 100 rows)
1000-row table, cap 100: rows returned 100, truncated true
exactly 100 matching rows, cap 100: rows returned 100, truncated false
7 matching rows, cap 100: rows returned 7, truncated false
a billion-row join, cap 100: rows returned 100, truncated true (the server stopped producing rows)
2 billion generated rows, cap 100: ERROR: canceling statement due to statement timeout (the row cap did not help here)
count(*) over 2 billion rows: ERROR: canceling statement due to statement timeout
stopped within 5 seconds of starting: true (role setting statement_timeout = 2s)
text-to-sql/output/05-role.txt
+12
View File
@@ -0,0 +1,12 @@
# What the t2s_reader role can and cannot do
SHOW statement_timeout ok: 2s
SHOW default_transaction_read_only ok: on
SELECT count(*) FROM orders ok: 1000
INSERT INTO orders ... (wall 1: read-only default) ERROR: cannot execute INSERT in a read-only transaction
SELECT set_config('default_transaction_read_only','off',false) ok: off
SHOW default_transaction_read_only ok: off
INSERT INTO orders ... (wall 2: no INSERT privilege) ERROR: permission denied for table orders
SELECT count(*) FROM api_keys ERROR: permission denied for table api_keys
SELECT email FROM customers ERROR: permission denied for table customers
SELECT id, name FROM customers LIMIT 1 ok: 1
text-to-sql/output/06-evaluation.txt
+13
View File
@@ -0,0 +1,13 @@
# Eight questions, scored three ways (the model is a script, not a language model)
question string execution what happened
How many customers are in Germany? no match same result
What is the total revenue of paid orders? no match same result
Which 3 products sold the most units? no match same result
Which customers have never ordered? no match same result
How many orders are there per status? no no DIFFERENT result: got [paid, 750], expected [paid, 250]
What is the average order value? no no DIFFERENT result: got [15.5000000000000000], expected [78.0150000000000000]
What is the revenue by country? no no refused by the guard: table sales is not allowed
How many orders were placed in March 2025? no match same result
string match: 0/8 execution match: 5/8 refused before running: 1 ran and returned a wrong answer: 2