apache / apache/iceberg

[Bug] REST server-side scan planning retains fetched FileScanTasks indefinitely

Open
#17,427 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Apache Iceberg version

1.10.1

### Query engine

Starrocks

### Please describe the bug 🐞

### Query engine

REST Catalog server-side scan planning.

### Please describe the bug

The reference implementation of REST server-side scan planning keeps all planned
`FileScanTask` objects in the singleton `InMemoryPlanningState`, even after clients
have successfully fetched every plan task.

As a result, repeated successful scans cause retained heap usage to grow
approximately linearly with the number of planned files.

The retained state is stored in the following maps:


Map> planTaskToFileScanTasks;
Map planTaskToNext;
Map asyncPlanningStates;

The normal successful fetch path only reads these maps:

`List fileScanTasks =
IN_MEMORY_PLANNING_STATE.fileScanTasksForPlanTask(planTask);

return FetchScanTasksResponse.builder()
.withFileScanTasks(fileScanTasks)
.withPlanTasks(IN_MEMORY_PLANNING_STATE.nextPlanTask(planTask))
.withSpecsById(table.specs())
.build();`

Neither fileScanTasksForPlanTask nor nextPlanTask removes the fetched entries.
The initial task returned by synchronous planTableScan or asynchronous
fetchPlanningResult is also retained.
Currently, the state is released only when cancelPlanning calls
InMemoryPlanningState.cancelPlan.
However, according to the REST Catalog OpenAPI specification, cancellation is not
required after scan tasks have been fetched for every plan task. This implies that a
successful fetch lifecycle must eventually release fetched task state without
requiring an explicit cancellation request.
Specification:
https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml
Relevant implementation:
https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/rest/CatalogHandlers.java
https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/rest/InMemoryPlanningState.java
Impact
In a long-running REST catalog deployment based on this implementation, we observed:

- Heap usage continuously increasing after successful scans

- Tens of millions of retained FileScanTask objects

- More than 20 GB of heap retained by planning state

- Full GC unable to reclaim the objects

- No active scans corresponding to most retained tasks

- No cancellation requests because all plan tasks had already been fetched successfully

In one observation window, approximately 25 million FileScanTask objects were
retained after about 22 hours of operation.
This may eventually cause excessive GC pressure or an OutOfMemoryError.

### Willingness to contribute

- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time

Contributor guide

Open the contributing guide

Research direction

Read core/src/main/java/org/apache/iceberg/rest/InMemoryPlanningState.java and trace its state access from CatalogHandlers.java, especially synchronous planning, fetch planning results, and fetching scan tasks. Run the relevant REST catalog tests if available; done means a successful fetch lifecycle no longer retains completed planning state, while cancellation still behaves correctly.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, openapi
Domain
api, backend, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
56/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.