`IcebergCatalog.validateNoLocationOverlap` is prone to data race conditions
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 522
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 137
Description
### Describe the bug
The design of the persistence infrastructure makes `validateNoLocationOverlap()` prone to false results.
With the right timing, it is possible to create multiple tables with the same base location. The flow is relatively simple and applies to tables and views and namespaces.
```
Time Create 'A' Create 'B'
==== ---------- ----------
0 create operation starts
1 create operation start
2 validateNoLocationOverlap("s3://foo/bar/")
--> OK (not used)
3 validateNoLocationOverlap("s3://foo/bar/")
--> OK (not used)
4 create entity
with base location `s3://foo/bar/`
5 create entity
with base location `s3://foo/bar/`
```
This is possible, because the `validateNoLocationOverlap()` performs 1 or many individual calls to `PolarisMetaStoreManager`, and each of those runs in within its own isolation (atomic or transactional). This means even serializable isolation would not help here.
The setting of `OPTIMIZED_SIBLING_CHECK` is irrelevant here, setting it may just make this issue harder to exploit (timing) in practice.
### To Reproduce
_No response_
### Actual Behavior
_No response_
### Expected Behavior
_No response_
### Additional context
_No response_
### System information
_No response_
Contributor guide
Research direction
Start by tracing IcebergCatalog.validateNoLocationOverlap and its calls to PolarisMetaStoreManager, then model the interleaving described for concurrent table, view, or namespace creation. Done means concurrent creates cannot both pass validation and use the same base location; add or run a regression test covering that timing if the existing test structure supports it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100