apache / apache/polaris

Polaris persistence mandates the use of transactions & serializable isolation

Open
#766 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
2.1k
Forks
522
Avg merge
1d 17h
Merged PRs (30d)
137

Description

### Describe the bug

Despite that `PolarisMetaStoreSession` [clearly says](https://github.com/apache/polaris/blob/88994f495a844f420187b9404511504601081a1a/polaris-core/src/main/java/org/apache/polaris/core/persistence/PolarisMetaStoreSession.java#L45-L46) that `it [is] really easy to back this using [...] simpler KV store.` the whole code architecture of everything around persistence in Polaris requires nothing less than (relational) transactions and strong consistency across multiple rows in multiple tables.

For example `org.apache.polaris.core.persistence.PolarisMetaStoreManagerImpl#writeEntity`:
```java
private void writeEntity(
@Nonnull PolarisMetaStoreSession ms,
@Nonnull PolarisBaseEntity entity,
boolean writeToActive) {
ms.writeToEntities(entity);
ms.writeToEntitiesChangeTracking(entity);

if (writeToActive) {
ms.writeToEntitiesActive(entity);
}
}
```
works against 2-3 different _rows_ in 2-3 different _tables_, requiring that all or none of the changes succeed - this is multiplied by the number of entities being written. On top it requires that entities being read before (existing and non-existing) did not change when the tx gets committed.

The "pattern" of having these different tables leaks into a lot of places. It effectively makes it rather impossible to use anything else than a relational database w/ isolation level `SERIALIZABLE`.

### To Reproduce

_No response_

### Actual Behavior

_No response_

### Expected Behavior

_No response_

### Additional context

_No response_

### System information

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with PolarisMetaStoreSession.java, especially the referenced lines, and inspect PolarisMetaStoreManagerImpl#writeEntity. Trace the writes to the entities, change-tracking, and active tables, then identify where transaction and SERIALIZABLE-isolation assumptions recur. Done means the persistence architecture no longer requires the relational multi-row transaction and strong-consistency pattern described in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.