apache / apache/polaris

`IcebergCatalog.validateNoLocationOverlap` is prone to data race conditions

Open
#2,295 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
2.1k
Forks
522
Avg merge
1d 22h
Merged PRs (30d)
137

Description

### Describe the bug

The design of the persistence infrastructure makes `validateNoLocationOverlap()` prone to false results.

With the right timing, it is possible to create multiple tables with the same base location. The flow is relatively simple and applies to tables and views and namespaces.

```
Time Create 'A' Create 'B'
==== ---------- ----------
0 create operation starts
1 create operation start
2 validateNoLocationOverlap("s3://foo/bar/")
--> OK (not used)
3 validateNoLocationOverlap("s3://foo/bar/")
--> OK (not used)
4 create entity
with base location `s3://foo/bar/`
5 create entity
with base location `s3://foo/bar/`
```

This is possible, because the `validateNoLocationOverlap()` performs 1 or many individual calls to `PolarisMetaStoreManager`, and each of those runs in within its own isolation (atomic or transactional). This means even serializable isolation would not help here.

The setting of `OPTIMIZED_SIBLING_CHECK` is irrelevant here, setting it may just make this issue harder to exploit (timing) in practice.

### To Reproduce

_No response_

### Actual Behavior

_No response_

### Expected Behavior

_No response_

### Additional context

_No response_

### System information

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by tracing IcebergCatalog.validateNoLocationOverlap and its calls to PolarisMetaStoreManager, then model the interleaving described for concurrent table, view, or namespace creation. Done means concurrent creates cannot both pass validation and use the same base location; add or run a regression test covering that timing if the existing test structure supports it.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.