Purge table task implementation prone to OOMs
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 522
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 137
Description
### Describe the bug
The implemented logic to prune Iceberg tables puts a huge amount of pressure to the Java heap.
The logic roughly works like this:
* Input parameter: pointer to a table metadata
* Read table metadata
* Read manifest lists of all snapshots
* Read all manifest files of all manifest lists
* Create a new task entity for each manifest file
Each manifest file can become really big, many MB of binary data.
The [base64 encoded full binary Iceberg manifest files](https://github.com/apache/polaris/blob/c9efc6c1af202686945efe2e19125e8f116a0206/runtime/service/src/main/java/org/apache/polaris/service/task/TableCleanupTaskHandler.java#L194) are included in the task entities, which are [consumed here](https://github.com/apache/polaris/blob/c9efc6c1af202686945efe2e19125e8f116a0206/runtime/service/src/main/java/org/apache/polaris/service/task/ManifestFileCleanupTaskHandler.java#L67).
Although some Java stream handling is being used, all manifest-files of the table to purge are [materialized at once on the Java heap](https://github.com/apache/polaris/blob/c9efc6c1af202686945efe2e19125e8f116a0206/runtime/service/src/main/java/org/apache/polaris/service/task/TableCleanupTaskHandler.java#L111-L130).
Even worse, the base64 encoded data is added to a JSON serialized object, which is added to a properties bag, which is in turn JSON serialized. Since JSON re-serializations of that property bag are "normal", the total heap pressure is bigger than the sum of all base64 serialized manifest-files.
### To Reproduce
_No response_
### Actual Behavior
_No response_
### Expected Behavior
_No response_
### Additional context
_No response_
### System information
_No response_
Contributor guide
Research direction
Start with TableCleanupTaskHandler.java, especially the manifest processing around lines 111-130 and 194, then inspect ManifestFileCleanupTaskHandler.java around line 67. Trace how manifest files become task entities and how their properties are serialized; done should prevent all manifest data for a table from being materialized and repeatedly JSON/base64 serialized on the heap.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100