apache / apache/datafusion

`TrackConsumersPool` makes memory updates wait on one shared lock

Open
#24,994 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

`TrackConsumersPool` uses one exclusive lock around its consumer map. Every `grow`, `shrink`, and `try_grow` call takes this lock, even though these operations only look up a consumer. This means memory updates from different tasks/partitions must wait for each other.

The values stored for each consumer are already safe to update from multiple threads. We should allow these lookups to happen at the same time while keeping registration and reporting exclusive.

### To Reproduce

N/A

### Expected behavior

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Locate TrackConsumersPool and inspect its consumer map plus the grow, shrink, and try_grow operations. Confirm that lookups can proceed concurrently while registration and reporting remain exclusive, then add or update tests that demonstrate the intended locking behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, performance
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.