AltimateAI / AltimateAI/altimate-code
[feat] Collapse entity-per-table warehouses in schema_index (composite digest)
- Vorherrschende Sprache
- TypeScript
- Sterne
- 811
- Forks
- 134
- Ø Merge
- 3 T. 2 Std.
- Gemergte PRs (30 T.)
- 50
Beschreibung
## Problem
When a database has many tables that share the same column structure — one ticker per table, one tenant per table, time-partitioned tables, per-region tables — the existing \`schema_index\` emits per-table schemas. N copies of the same schema fill the agent's context with redundancy. With \`schema_index\`'s truncation cap, the agent sometimes can't see all tables at all.
Concrete example we hit on a benchmark workload:
- stockmarket dataset has **2754 ticker tables**, every one with identical columns (\`Date\`, \`Open\`, \`High\`, \`Low\`, \`Close\`, \`Adj Close\`, \`CloseUSD\`).
- \`schema_index\` truncates to 200 → agent only sees ticker symbols A–Bxxx and has to discover the rest at runtime via SQL probes.
- Score on this dataset: 1/5 vs. yelp (per-table-is-different schemas) at 6/7. Direct attribution to redundant per-table schemas filling context.
## Proposed mechanism
Heuristic in \`schema_index\`'s scan logic, after enumerating tables for a connection:
1. **Group tables by column-structure fingerprint** — sorted tuple of \`(column_name, column_type)\`.
2. **Detect entity-per-table pattern** — if any single fingerprint group covers ≥50% of tables AND has ≥20 tables, mark that group as the \"entity\" group.
3. **Emit a different digest format** for the entity group:
\`\`\`
schema: .
pattern: entity-per-table
table_count: 2754
composite_columns: [{name: \"Date\", type: \"VARCHAR\"}, {name: \"Open\", type: \"DOUBLE\"}, …]
sample_table: AAPL
table_names: [AAPL, AAA, AAL, AAN, …] ← full list, searchable
\`\`\`
4. Tables NOT in the entity group still get per-table schemas. The two formats coexist.
5. \`schema_search\` should still surface table names from entity groups in lookups.
## What it enables
- ~10× context-saving on entity-per-table warehouses.
- Agent sees ALL tables (full list of names) instead of a truncated subset.
- Per-table queries still work — composite schema describes the columns; the agent uses table names directly.
- Backwards compatible: tables with unique structures still get per-table schemas as today.
## Beyond benchmark / general use cases
This is a real production pattern, common in:
- Multi-tenant SaaS (Stripe-style \"one table per customer for isolation\")
- Time-partitioned analytics (\`events_2024_01_01\`, \`events_2024_01_02\`, …)
- Per-region tables (\`orders_us\`, \`orders_eu\`, \`orders_apac\`)
- Per-environment (dev/staging/prod schemas with same tables)
- Per-source ingestion (\`events_segment\`, \`events_amplitude\`, \`events_mixpanel\`)
- Sharded analytics (per-customer, per-tenant)
Customers using altimate-code on these warehouses today are losing context capacity to redundant per-table schemas they could collapse to one composite + a name list.
## Implementation pointers
- Heuristic detector: new helper in \`packages/opencode/src/altimate/native/schema/cache.ts\`
- Tool wrapper: \`packages/opencode/src/altimate/tools/schema-index.ts\` (or wherever it lives)
- Output format reference: \`packages/opencode/src/altimate/tools/schema-inspect.ts\`
- Estimated scope: ~150-200 LOC + tests
## Tests to add
- Detector unit tests:
- 100 same-shape tables → marked entity
- 5 tables of various shapes → not marked
- 50 same-shape + 50 different-shape → not marked (under 50% threshold)
- Integration test against in-memory SQLite fixture with 30 identical-schema tables
- Backwards-compat test: existing single-table warehouse output unchanged
Beitragsleitfaden
Bewertung
Dieses Issue wurde noch nicht bewertet.