[Bug] REST server-side scan planning retains fetched FileScanTasks indefinitely
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Apache Iceberg version
1.10.1
### Query engine
Starrocks
### Please describe the bug 🐞
### Query engine
REST Catalog server-side scan planning.
### Please describe the bug
The reference implementation of REST server-side scan planning keeps all planned
`FileScanTask` objects in the singleton `InMemoryPlanningState`, even after clients
have successfully fetched every plan task.
As a result, repeated successful scans cause retained heap usage to grow
approximately linearly with the number of planned files.
The retained state is stored in the following maps:
Map> planTaskToFileScanTasks;
Map planTaskToNext;
Map asyncPlanningStates;
The normal successful fetch path only reads these maps:
`List fileScanTasks =
IN_MEMORY_PLANNING_STATE.fileScanTasksForPlanTask(planTask);
return FetchScanTasksResponse.builder()
.withFileScanTasks(fileScanTasks)
.withPlanTasks(IN_MEMORY_PLANNING_STATE.nextPlanTask(planTask))
.withSpecsById(table.specs())
.build();`
Neither fileScanTasksForPlanTask nor nextPlanTask removes the fetched entries.
The initial task returned by synchronous planTableScan or asynchronous
fetchPlanningResult is also retained.
Currently, the state is released only when cancelPlanning calls
InMemoryPlanningState.cancelPlan.
However, according to the REST Catalog OpenAPI specification, cancellation is not
required after scan tasks have been fetched for every plan task. This implies that a
successful fetch lifecycle must eventually release fetched task state without
requiring an explicit cancellation request.
Specification:
https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml
Relevant implementation:
https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/rest/CatalogHandlers.java
https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/rest/InMemoryPlanningState.java
Impact
In a long-running REST catalog deployment based on this implementation, we observed:
- Heap usage continuously increasing after successful scans
- Tens of millions of retained FileScanTask objects
- More than 20 GB of heap retained by planning state
- Full GC unable to reclaim the objects
- No active scans corresponding to most retained tasks
- No cancellation requests because all plan tasks had already been fetched successfully
In one observation window, approximately 25 million FileScanTask objects were
retained after about 22 hours of operation.
This may eventually cause excessive GC pressure or an OutOfMemoryError.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
Contributor guide
Research direction
Read core/src/main/java/org/apache/iceberg/rest/InMemoryPlanningState.java and trace its state access from CatalogHandlers.java, especially synchronous planning, fetch planning results, and fetching scan tasks. Run the relevant REST catalog tests if available; done means a successful fetch lifecycle no longer retains completed planning state, while cancellation still behaves correctly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, openapi
- Domain
- api, backend, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 56/100