[Bug report] Schema cascade can race with child metadata writes and leave active orphan rows
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 935
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 298
Description
### Version
main branch
### Describe what's wrong
Schema cascade deletion is not fully coordinated with concurrent child metadata writes across Gravitino server instances.
The existing `TreeLock` protects these operations only within one process. At the relational storage layer, only some child inserts and cross-schema table moves currently acquire a shared lock on the parent schema row. Existing entity updates and model-version writes can therefore interleave with a schema cascade after descendant cleanup has started.
Known unsafe paths include:
- `FunctionMetaService.updateFunction` inserts a new function-version row and ignores a zero-row metadata update.
- A same-schema `TableMetaService.updateTable` can write a new table-version row, commit the transaction, and only then report the zero-row metadata update as a failure.
- A cross-schema table move protects only the destination schema, not the source schema.
- `ModelVersionMetaService.insertModelVersion` and `updateModelVersion` do not participate in the schema locking protocol.
- Other table, view, fileset, topic, function, and model update paths do not consistently validate parent-schema liveness inside their write transactions.
As a result, a schema cascade can commit while an overlapping writer leaves an active row in a child or version table whose schema or parent entity has already been deleted.
This is a pre-existing cross-instance concurrency problem. It was identified while reviewing #12350, but it is outside #12342's metalake/catalog/schema OCC scope and should be fixed separately.
### Error message and/or stacktrace
The failure can be silent. Depending on the write path, the caller may receive an `IOException`, a not-found error, or even success while the database contains an active orphan row.
For example, `updateTable` can commit a new active table-version row and then throw an `IOException` because its table metadata CAS affected zero rows.
### How to reproduce
One deterministic example is:
1. Create a schema and a table in that schema.
2. Start a table update and pause it after reading the existing table metadata.
3. Cascade-delete the schema and commit the deletion.
4. Resume the table update.
5. The table metadata update affects zero rows, but the transaction still writes and commits a new table-version row.
6. The caller receives an update failure while an active orphan version remains in the database.
A similar race exists for function updates. Model-version insertion can interleave between the cascade's model-version cleanup and model metadata deletion.
### Additional context
Implement a complete database-backed schema-to-child write protocol:
- Acquire a shared lock on the active parent schema row as the first operation in every child create, update, and version-write transaction.
- For cross-schema table moves, validate and lock both the source and destination schemas.
- Cover table, view, fileset, topic, function, model, and model-version write paths.
- Treat a zero-row child CAS/update as a transaction failure before any transaction commit, so version rows, aliases, columns, and change-log writes are rolled back.
- Preserve existing idempotent behavior for pure delete and retention-cleanup operations that cannot create or reactivate child rows.
- Add deterministic service-level concurrency tests using the real write paths rather than invoking the schema-lock helper directly.
- Test both race orderings:
- the writer obtains the shared schema lock first, causing the cascade to wait;
- the cascade wins first, causing the writer to fail without leaving active descendants.
- Assert that no affected metadata, version, alias, or column table contains an active (`deleted_at = 0`) orphan row.
- Run the concurrency tests against H2, MySQL, and PostgreSQL.
The namespace OCC work remains in #12350. Catalog wrapper and classloader lifetime management remains tracked separately by #12403.
Contributor guide
Assessment
This issue has not been assessed yet.