apache / apache/polaris

Purge table task implementation prone to OOMs

Open
#2,365 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
2.1k
Forks
522
Avg merge
1d 22h
Merged PRs (30d)
137

Description

### Describe the bug

The implemented logic to prune Iceberg tables puts a huge amount of pressure to the Java heap.

The logic roughly works like this:
* Input parameter: pointer to a table metadata
* Read table metadata
* Read manifest lists of all snapshots
* Read all manifest files of all manifest lists
* Create a new task entity for each manifest file

Each manifest file can become really big, many MB of binary data.

The [base64 encoded full binary Iceberg manifest files](https://github.com/apache/polaris/blob/c9efc6c1af202686945efe2e19125e8f116a0206/runtime/service/src/main/java/org/apache/polaris/service/task/TableCleanupTaskHandler.java#L194) are included in the task entities, which are [consumed here](https://github.com/apache/polaris/blob/c9efc6c1af202686945efe2e19125e8f116a0206/runtime/service/src/main/java/org/apache/polaris/service/task/ManifestFileCleanupTaskHandler.java#L67).

Although some Java stream handling is being used, all manifest-files of the table to purge are [materialized at once on the Java heap](https://github.com/apache/polaris/blob/c9efc6c1af202686945efe2e19125e8f116a0206/runtime/service/src/main/java/org/apache/polaris/service/task/TableCleanupTaskHandler.java#L111-L130).

Even worse, the base64 encoded data is added to a JSON serialized object, which is added to a properties bag, which is in turn JSON serialized. Since JSON re-serializations of that property bag are "normal", the total heap pressure is bigger than the sum of all base64 serialized manifest-files.

### To Reproduce

_No response_

### Actual Behavior

_No response_

### Expected Behavior

_No response_

### Additional context

_No response_

### System information

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with TableCleanupTaskHandler.java, especially the manifest processing around lines 111-130 and 194, then inspect ManifestFileCleanupTaskHandler.java around line 67. Trace how manifest files become task entities and how their properties are serialized; done should prevent all manifest data for a table from being materialized and repeatedly JSON/base64 serialized on the heap.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.