Cubestore: DROP TABLE via MySQL wire fails with 'One value expected in SchemaRocksTable for X but nothing found' — schema name index out of sync with information_schema
- Dominant language
- Rust
- Stars
- 20.8k
- Forks
- 2.1k
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 181
Description
## Summary
On Cubestore v1.6.44, running `DROP TABLE .` via the MySQL wire (port 3306) fails with the following error for a subset of schemas:
```
Internal: One value expected in SchemaRocksTable for "SEMANTIC_LAYER_LATESTLABS_TESTING112" but nothing found
```
The affected schemas' tables **still appear in `information_schema.tables`** via the same MySQL client, but any DROP against them fails at this metadata lookup. Cube's own driver (WebSocket transport on port 3030) does not surface this error because its `dropOrphanedTables` iterates Cube's own `version_entries` (which appear to also be missing these schemas), so it never attempts a DROP on the affected rows — meanwhile the physical tables remain visible via `information_schema.tables` and accumulate.
## Environment
- Cube.js: v1.6.44 (self-hosted, `service-semantic-layer` deployment)
- Cubestore router: separate pod, WebSocket transport for Cube's driver + MySQL wire on :3306 for external clients
- Client: `mysql2/promise` from Node.js
## Reproduction
1. `SELECT COUNT(*) FROM information_schema.tables WHERE table_schema = 'SEMANTIC_LAYER_LATESTLABS_TESTING112'` — returns > 0 rows (e.g. 63 in our case).
2. `DROP TABLE \`SEMANTIC_LAYER_LATESTLABS_TESTING112\`.\`sales_sales_rollup_zhtt0ebz_phttqxd5_1kledcg\`` — fails with `Internal: One value expected in SchemaRocksTable for "SEMANTIC_LAYER_LATESTLABS_TESTING112" but nothing found`.
3. Same DROP via Cube's own `@cubejs-backend/cubestore-driver` on the WebSocket transport — never fires, because Cube's driver's list step doesn't return these tables (RocksDB `version_entries` are missing).
## Root cause (from source)
Error is raised in [`rust/cubestore/cubestore/src/metastore/rocks_table.rs`](https://github.com/cube-js/cube/blob/v1.6.44/rust/cubestore/cubestore/src/metastore/rocks_table.rs) at `get_single_row_by_index`:
```rust
fn get_single_row_by_index(...) -> Result, CubeError> {
let row = self.get_row_by_index_opt(row_key, secondary_index, false)?;
row.ok_or(CubeError::internal(format!(
"One value expected in {:?} for {:?} but nothing found",
self, row_key
)))
}
```
Condition: `get_row_ids_by_index` returns an empty vec for the given secondary key — the `SchemaRocksIndex::Name` secondary index has no entry for the requested schema name, even though tables referencing that schema exist in physical storage.
Callsites in [`metastore/mod.rs`](https://github.com/cube-js/cube/blob/v1.6.44/rust/cubestore/cubestore/src/metastore/mod.rs):
- `get_table_by_name` (~L1926) — called by the MySQL wire's DROP TABLE handler.
- `get_schema` (~L1945)
- `create_table` (L2230-ish, in `drop_if_exists` path)
## Related prior fix
Commit [`b74c072`](https://github.com/cube-js/cube/commit/b74c0721fb8d62d9e52570a3d5b883604b00648c) (Dec 2022) fixed a sibling error on the same `SchemaRocksTable` — `"Row with id X is not found for SchemaRocksTable"` — by sorting WAL log filenames numerically before replay during metastore loading from dump. That fix addressed a different code path (`get_row_by_index_opt` returning a stale row-id pointing to a missing primary row) but is in the same family: RocksDB metastore secondary/primary index inconsistency after replay.
## What we've tried
- Cubestore router restart: does not clear the inconsistency (metastore restored from S3 snapshot, presumably including the corrupted state).
- Querying via Cube's own WebSocket driver: bypasses the error (doesn't hit `get_table_by_name` for affected schemas) but also doesn't clean up the zombie tables.
## Impact
Any tenant with orphan pre-aggregation tables in an affected schema cannot be cleaned via external tooling using the MySQL wire. `information_schema.tables` continues to report the tables, accumulating storage without a cleanup path short of manual RocksDB write or metastore snapshot restore.
## Ask
- Is this a known issue with a documented recovery procedure (RocksDB compaction, metastore repair tool, etc.)?
- If not — would you accept a bug report with a `cubestore-dumps/` S3 snapshot for diagnosis? Happy to provide.
- Ideally: a self-heal mechanism analogous to the `Row exists in secondary index however missing in ...` "Repairing index" path in the same file (line ~833 in v1.6.44), but applied when a secondary index is empty despite the primary having tables that reference the schema.
## Search precedent
- No filed issues match this exact string ([search](https://github.com/cube-js/cube/issues?q=%22One+value+expected+in+SchemaRocksTable%22)).
- No Stack Overflow / community forum hits.
Happy to help with reproduction / diagnostics.
Contributor guide
Research direction
Start in rust/cubestore/cubestore/src/metastore/rocks_table.rs at get_single_row_by_index and the existing index-repair path near line 833. Then trace get_table_by_name, get_schema, and create_table in metastore/mod.rs and reproduce the MySQL DROP against a schema present in information_schema.tables. Done should include a confirmed recovery or repair path that lets affected tables be cleaned without manual RocksDB writes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- mysql, rust
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 38/100