cube-js / cube-js/cube

Cubestore: DROP TABLE via MySQL wire fails with 'One value expected in SchemaRocksTable for X but nothing found' — schema name index out of sync with information_schema

Open
#11,212 0 comments 0 reactions 0 assignees View on GitHub
cube store
Dominant language
Rust
Stars
20.8k
Forks
2.1k
Avg merge
1d 2h
Merged PRs (30d)
181

Description

## Summary

On Cubestore v1.6.44, running `DROP TABLE .` via the MySQL wire (port 3306) fails with the following error for a subset of schemas:

```
Internal: One value expected in SchemaRocksTable for "SEMANTIC_LAYER_LATESTLABS_TESTING112" but nothing found
```

The affected schemas' tables **still appear in `information_schema.tables`** via the same MySQL client, but any DROP against them fails at this metadata lookup. Cube's own driver (WebSocket transport on port 3030) does not surface this error because its `dropOrphanedTables` iterates Cube's own `version_entries` (which appear to also be missing these schemas), so it never attempts a DROP on the affected rows — meanwhile the physical tables remain visible via `information_schema.tables` and accumulate.

## Environment

- Cube.js: v1.6.44 (self-hosted, `service-semantic-layer` deployment)
- Cubestore router: separate pod, WebSocket transport for Cube's driver + MySQL wire on :3306 for external clients
- Client: `mysql2/promise` from Node.js

## Reproduction

1. `SELECT COUNT(*) FROM information_schema.tables WHERE table_schema = 'SEMANTIC_LAYER_LATESTLABS_TESTING112'` — returns > 0 rows (e.g. 63 in our case).
2. `DROP TABLE \`SEMANTIC_LAYER_LATESTLABS_TESTING112\`.\`sales_sales_rollup_zhtt0ebz_phttqxd5_1kledcg\`` — fails with `Internal: One value expected in SchemaRocksTable for "SEMANTIC_LAYER_LATESTLABS_TESTING112" but nothing found`.
3. Same DROP via Cube's own `@cubejs-backend/cubestore-driver` on the WebSocket transport — never fires, because Cube's driver's list step doesn't return these tables (RocksDB `version_entries` are missing).

## Root cause (from source)

Error is raised in [`rust/cubestore/cubestore/src/metastore/rocks_table.rs`](https://github.com/cube-js/cube/blob/v1.6.44/rust/cubestore/cubestore/src/metastore/rocks_table.rs) at `get_single_row_by_index`:

```rust
fn get_single_row_by_index(...) -> Result, CubeError> {
let row = self.get_row_by_index_opt(row_key, secondary_index, false)?;
row.ok_or(CubeError::internal(format!(
"One value expected in {:?} for {:?} but nothing found",
self, row_key
)))
}
```

Condition: `get_row_ids_by_index` returns an empty vec for the given secondary key — the `SchemaRocksIndex::Name` secondary index has no entry for the requested schema name, even though tables referencing that schema exist in physical storage.

Callsites in [`metastore/mod.rs`](https://github.com/cube-js/cube/blob/v1.6.44/rust/cubestore/cubestore/src/metastore/mod.rs):
- `get_table_by_name` (~L1926) — called by the MySQL wire's DROP TABLE handler.
- `get_schema` (~L1945)
- `create_table` (L2230-ish, in `drop_if_exists` path)

## Related prior fix

Commit [`b74c072`](https://github.com/cube-js/cube/commit/b74c0721fb8d62d9e52570a3d5b883604b00648c) (Dec 2022) fixed a sibling error on the same `SchemaRocksTable` — `"Row with id X is not found for SchemaRocksTable"` — by sorting WAL log filenames numerically before replay during metastore loading from dump. That fix addressed a different code path (`get_row_by_index_opt` returning a stale row-id pointing to a missing primary row) but is in the same family: RocksDB metastore secondary/primary index inconsistency after replay.

## What we've tried

- Cubestore router restart: does not clear the inconsistency (metastore restored from S3 snapshot, presumably including the corrupted state).
- Querying via Cube's own WebSocket driver: bypasses the error (doesn't hit `get_table_by_name` for affected schemas) but also doesn't clean up the zombie tables.

## Impact

Any tenant with orphan pre-aggregation tables in an affected schema cannot be cleaned via external tooling using the MySQL wire. `information_schema.tables` continues to report the tables, accumulating storage without a cleanup path short of manual RocksDB write or metastore snapshot restore.

## Ask

- Is this a known issue with a documented recovery procedure (RocksDB compaction, metastore repair tool, etc.)?
- If not — would you accept a bug report with a `cubestore-dumps/` S3 snapshot for diagnosis? Happy to provide.
- Ideally: a self-heal mechanism analogous to the `Row exists in secondary index however missing in ...` "Repairing index" path in the same file (line ~833 in v1.6.44), but applied when a secondary index is empty despite the primary having tables that reference the schema.

## Search precedent

- No filed issues match this exact string ([search](https://github.com/cube-js/cube/issues?q=%22One+value+expected+in+SchemaRocksTable%22)).
- No Stack Overflow / community forum hits.

Happy to help with reproduction / diagnostics.

Contributor guide

Open the contributing guide

Research direction

Start in rust/cubestore/cubestore/src/metastore/rocks_table.rs at get_single_row_by_index and the existing index-repair path near line 833. Then trace get_table_by_name, get_schema, and create_table in metastore/mod.rs and reproduce the MySQL DROP against a schema present in information_schema.tables. Done should include a confirmed recovery or repair path that lets affected tables be cleaned without manual RocksDB writes.

Written by the indexing model from the issue text.

Assessment

Tech stack
mysql, rust
Domain
backend, databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.