Polaris persistence mandates the use of transactions & serializable isolation
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 522
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 137
Description
### Describe the bug
Despite that `PolarisMetaStoreSession` [clearly says](https://github.com/apache/polaris/blob/88994f495a844f420187b9404511504601081a1a/polaris-core/src/main/java/org/apache/polaris/core/persistence/PolarisMetaStoreSession.java#L45-L46) that `it [is] really easy to back this using [...] simpler KV store.` the whole code architecture of everything around persistence in Polaris requires nothing less than (relational) transactions and strong consistency across multiple rows in multiple tables.
For example `org.apache.polaris.core.persistence.PolarisMetaStoreManagerImpl#writeEntity`:
```java
private void writeEntity(
@Nonnull PolarisMetaStoreSession ms,
@Nonnull PolarisBaseEntity entity,
boolean writeToActive) {
ms.writeToEntities(entity);
ms.writeToEntitiesChangeTracking(entity);
if (writeToActive) {
ms.writeToEntitiesActive(entity);
}
}
```
works against 2-3 different _rows_ in 2-3 different _tables_, requiring that all or none of the changes succeed - this is multiplied by the number of entities being written. On top it requires that entities being read before (existing and non-existing) did not change when the tx gets committed.
The "pattern" of having these different tables leaks into a lot of places. It effectively makes it rather impossible to use anything else than a relational database w/ isolation level `SERIALIZABLE`.
### To Reproduce
_No response_
### Actual Behavior
_No response_
### Expected Behavior
_No response_
### Additional context
_No response_
### System information
_No response_
Contributor guide
Research direction
Start with PolarisMetaStoreSession.java, especially the referenced lines, and inspect PolarisMetaStoreManagerImpl#writeEntity. Trace the writes to the entities, change-tracking, and active tables, then identify where transaction and SERIALIZABLE-isolation assumptions recur. Done means the persistence architecture no longer requires the relational multi-row transaction and strong-consistency pattern described in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100