Stop pasting your database schema into the prompt

The obvious way to make an LLM answer questions about your database is to put the schema in the prompt — every table, every column — and let it write queries. It works in the demo. Then you connect a real ERP with 500 tables and it falls apart on three axes at once: latency (huge prompts are slow), cost (you pay for the schema on every single question), and accuracy (the model drowns in tables it will never need).

OpenNeko answers questions over customer databases it has never seen, through GraphJin (a GraphQL layer over Postgres/MySQL), with an agent that runs in a sandbox and discovers the schema itself. Getting that to hand-tuned speed took four distinct mechanisms. Here's the shape of the whole pipeline, then each mechanism with the actual code.

1. Prefetch discovery to disk — and leave the agent a README

Broad schema discovery is expensive and, crucially, the same every time. So the worker prefetches it once per boot: four JSON files pulled from GraphJin's discovery endpoints and written into the agent's workspace — tables.json (every table, with schema/type/column counts), namespaces.json, insights.json (hub tables, hot relationships, precomputed join paths, query templates, data-quality flags), and syntax.json (the full query-DSL reference).

Next to them goes an INDEX.md — written to the agent, as instructions (packages/llm/src/knowledge-pack.ts):

This directory holds JSON files prefetched from the running GraphJin
server's HTTP discovery endpoints, plus this index. The four JSONs are
regenerated on every worker boot and on demand. Do NOT run
`graphjin cli list_tables` / `describe_table` / `get_query_syntax` /
`get_schema_insights` — those broad discovery dumps are already on disk
in the JSONs below.
 
For targeted relationship questions, these read-only CLI tools are allowed:
 
- `graphjin cli find_path --args '{"from_table":"…","to_table":"…"}'`
  — exact relationship path between two specific tables.
- `graphjin cli explore_relationships --args '{"table":"…"}'`
  — connected tables around one focal table.

The pattern generalizes beyond databases: treat the filesystem as a pre-warmed cache for tool calls, and document the cache in the agent's own instruction language. The agent reads insights.json when planning a multi-table query the way you'd check a map — on demand, not memorized. Only two narrow tools stay live, for questions the dump can't answer (shortest join path between two specific tables).

Result: zero broad discovery calls at question time, and the schema never enters the prompt at all.

2. Warm-start each role from a few entrypoints

A CFO asking "how are margins trending?" and a COO asking "what's stuck in fulfillment?" need different corners of the same database. So the prompt gets a role-shaped warm start derived from insights.json: up to 12 hub tables, a few seed searches for that role, up to 8 curated query patterns (packages/llm/src/discovery-pathways.ts):

const ROLE_SEED_SEARCHES: Record<string, string[]> = {
  CFO: ["margin by product", "accounts receivable aging", "cost trends"],
  COO: ["order fulfillment lead time", "inventory below reorder point", "throughput"],
  // …CEO, CRO, CMO, CIO, CPO
};

which renders into the prompt as a compact block:

<discovery-pathways>
Start discovery from these hub tables (inspect them before searching wider):
- sales_order_header
- product_inventory
Seed searches worth running first for this role:
- "inventory below reorder point"
Curated query patterns (prefer adapting one over writing from scratch):
- Orders stuck in pending — join header→detail, filter status/age
</discovery-pathways>

The design rule is in the source, and it's the whole philosophy in one line:

/**
 * The prompt block: a warm-start, not a schema dump. Tells the agent
 * where to START discovering, never what every table looks like.
 */

The pathways are cached per (org, role, intent) with a one-hour TTL, so parallel briefing-card builds for the same role share one computation instead of recomputing per card. And when insights.json is missing or malformed, the block degrades to role seeds only — a worse warm start, never a crash.

3. Bound the self-correction loop — and know what not to retry

Agents writing queries against unfamiliar schemas will sometimes produce output that parses wrong or comes back empty. Failing the job is wasteful; shipping the broken answer is worse. The middle path is a validated turn (packages/llm/src/agent-validate-loop.ts):

/**
 * A job agent's reply that fails validation no longer fails the whole
 * job: the validation error (and the tail of the rejected reply) is fed
 * back to the agent for a corrective attempt, bounded by maxAttempts.
 * Backend failures and upstream provider outages still throw
 * immediately — retrying those wastes spend.
 */
export async function runValidatedAgentTurn<T>(
  opts: ValidatedAgentTurnOptions<T>,
): Promise<{ value: T; finalText: string; attempts: number }>;

Every job defines a validate(finalText) that throws a precise, agent-readable error. On failure, the error plus the tail of the rejected output goes back to the agent as the next turn's input: here's what you produced, here's exactly why it's unusable. Default budget: 3 attempts total.

The distinction in the comment is the part that saves real money: a validation failure means the agent was wrong — retry it. A provider outage means the world is wrong — retrying burns tokens against a wall, so it throws immediately. Conflating those two failure modes is an easy way to pay for the same outage three times.

4. Compact tool output — losslessly, and measure before you trust it

The dominant token sink in the loop isn't the prompt — it's query results flowing back into context. Ask for 200 rows and JSON repeats every column key 200 times:

[
  {"region": "West", "revenue": 4120, "orders": 310},
  {"region": "East", "revenue": 3890, "orders": 287}
]

The fix is columnar rewriting — state the keys once:

{"__neko_cols__": {"cols": ["region", "revenue", "orders"],
 "rows": [["West", 4120, 310], ["East", 3890, 287]]}}

For wide result sets this routinely halves the byte count, and the guarantees are strict (packages/llm/src/work/tool-output/compact-json.ts):

/**
 * If `arr` is an array of >= MIN_ROWS plain objects that all share the
 * EXACT same key set and whose every value is a primitive, return its
 * columnar envelope. Otherwise return null. The exact-key-set
 * requirement is what keeps the transform unambiguously reversible: no
 * "missing key vs null value" guessing on expand.
 */

Lossless by construction: expandJson(parse(compactJson(raw).text)) deep-equals the original for every transformed input, ragged or nested shapes pass through minified, and if the compact form isn't smaller the original is returned untouched — the transform can never regress.

And one engineering habit worth stealing: this module currently runs in measurement mode. It's wired to metrics that record how much it would save on real workloads, but it does not yet rewrite what the model reads — the columnar envelope needs a one-line legend first, and when the rows are someone's financial data, "should be fine" is not a deploy strategy. Measure, then flip the switch.

Where this lands

Prefetched discovery, role-shaped entrypoints, a bounded correction loop, and smaller result payloads compound: agentic answers over a never-before-seen schema now run at the speed of the queries we used to tune by hand — with no fine-tuning and no training on your schema, which means it still works when your schema changes tomorrow.

All of it is open — github.com/open-neko/openneko — and runs on your infrastructure with whichever model you rent this quarter. That split is the idea OpenNeko is built on: rent the intelligence, own the context.