apache / apache/iceberg

Race Condition in CachingCatalog

Open
#17,338 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Apache Iceberg version

1.10.0

### Query engine

Spark

### Please describe the bug 🐞

Reopening this closed issue https://github.com/apache/iceberg/issues/10493

CachingCatalog is susceptible to a race condition because it uses Cache.put. Cache.put inherently races with Cache.invalidate / Cache.invalidateAll

thread A: check cache - empty
thread A: compute new value for the cache (not yet stale)
thread B: change external state (value computed by A becomes stale at this point
thread B: invalidate() or invalidateAll()
thread A: put into the cache (the put value is stale)
stale value remains visible in the cache until cache expiration

This realizes when enabling OpenLineage which calls `loadTable()` in a background thread and when that happens while write is in progress, the writer thread reading back the same table it wrote to ends up reading stale cache reference from the loadTable() call -- this has caused a major production incident on our end.

I believe the solution to this is to expire cache on commit - this should be the correct behavior, the table state has changed with the commit so any cache references to this table should be expired.

### Willingness to contribute

- [x] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time

Contributor guide

Open the contributing guide

Research direction

Start by tracing CachingCatalog's loadTable cache path, including Cache.put, invalidate/invalidateAll, and the commit path. Reproduce or test the race where a value computed before a commit is inserted afterward; done means committed table state cannot leave a stale cache reference visible.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.