AltimateAI / AltimateAI/altimate-code

[feat] Collapse entity-per-table warehouses in schema_index (composite digest)

Abierto
#760 0 comentarios 0 reacciones 0 asignados Ver en GitHub
enhancement priority:medium sql-engine warehouse
Lenguaje dominante
TypeScript
Estrellas
811
Forks
134
Merge medio
3 d 2 h
PR fusionados (30 d)
50

Descripción

## Problem

When a database has many tables that share the same column structure — one ticker per table, one tenant per table, time-partitioned tables, per-region tables — the existing \`schema_index\` emits per-table schemas. N copies of the same schema fill the agent's context with redundancy. With \`schema_index\`'s truncation cap, the agent sometimes can't see all tables at all.

Concrete example we hit on a benchmark workload:

- stockmarket dataset has **2754 ticker tables**, every one with identical columns (\`Date\`, \`Open\`, \`High\`, \`Low\`, \`Close\`, \`Adj Close\`, \`CloseUSD\`).
- \`schema_index\` truncates to 200 → agent only sees ticker symbols A–Bxxx and has to discover the rest at runtime via SQL probes.
- Score on this dataset: 1/5 vs. yelp (per-table-is-different schemas) at 6/7. Direct attribution to redundant per-table schemas filling context.

## Proposed mechanism

Heuristic in \`schema_index\`'s scan logic, after enumerating tables for a connection:

1. **Group tables by column-structure fingerprint** — sorted tuple of \`(column_name, column_type)\`.
2. **Detect entity-per-table pattern** — if any single fingerprint group covers ≥50% of tables AND has ≥20 tables, mark that group as the \"entity\" group.
3. **Emit a different digest format** for the entity group:

\`\`\`
schema: .
pattern: entity-per-table
table_count: 2754
composite_columns: [{name: \"Date\", type: \"VARCHAR\"}, {name: \"Open\", type: \"DOUBLE\"}, …]
sample_table: AAPL
table_names: [AAPL, AAA, AAL, AAN, …] ← full list, searchable
\`\`\`

4. Tables NOT in the entity group still get per-table schemas. The two formats coexist.
5. \`schema_search\` should still surface table names from entity groups in lookups.

## What it enables

- ~10× context-saving on entity-per-table warehouses.
- Agent sees ALL tables (full list of names) instead of a truncated subset.
- Per-table queries still work — composite schema describes the columns; the agent uses table names directly.
- Backwards compatible: tables with unique structures still get per-table schemas as today.

## Beyond benchmark / general use cases

This is a real production pattern, common in:

- Multi-tenant SaaS (Stripe-style \"one table per customer for isolation\")
- Time-partitioned analytics (\`events_2024_01_01\`, \`events_2024_01_02\`, …)
- Per-region tables (\`orders_us\`, \`orders_eu\`, \`orders_apac\`)
- Per-environment (dev/staging/prod schemas with same tables)
- Per-source ingestion (\`events_segment\`, \`events_amplitude\`, \`events_mixpanel\`)
- Sharded analytics (per-customer, per-tenant)

Customers using altimate-code on these warehouses today are losing context capacity to redundant per-table schemas they could collapse to one composite + a name list.

## Implementation pointers

- Heuristic detector: new helper in \`packages/opencode/src/altimate/native/schema/cache.ts\`
- Tool wrapper: \`packages/opencode/src/altimate/tools/schema-index.ts\` (or wherever it lives)
- Output format reference: \`packages/opencode/src/altimate/tools/schema-inspect.ts\`
- Estimated scope: ~150-200 LOC + tests

## Tests to add

- Detector unit tests:
- 100 same-shape tables → marked entity
- 5 tables of various shapes → not marked
- 50 same-shape + 50 different-shape → not marked (under 50% threshold)
- Integration test against in-memory SQLite fixture with 30 identical-schema tables
- Backwards-compat test: existing single-table warehouse output unchanged

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.