Excessive Memory Usage by LookupCoordinatorManager in master nodes Due to keeping a copy of each Lookups per process
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
Excessive Memory Usage by LookupCoordinatorManager in master nodes Due to keeping a copy of each Lookups per process
### Affected Version
The Druid version where the problem was encountered is 24.0.0
### Description
I encountered a severe memory usage issue in master nodes when using lookups. The [LookupCoordinatorManager](https://github.com/apache/druid/blob/24.0.0/server/src/main/java/org/apache/druid/server/lookup/cache/LookupCoordinatorManager.java) keeps **a copy** of each static lookup in the member `knownOldState` for **each running process on query and data servers** (historical, middleManager, peon, broker, router) in the cluster. As a result, the memory consumption scales **linearly** with the number of processes, causing excessive heap usage on master nodes and `ExitOnOutOfMemoryError` crashes.
## Observed Behavior
- I added lookups with a **total size of 100MB**.
- My cluster has **at least running 14 processes** on data and query servers without counting peon processes.
- The lookup memory consumption on the master node reached more than **3.6GB** (`100MB x 2 x 14`), leading to OOM java exception
- Heap dump analysis using **Eclipse Memory Analyzer (MAT)** showed that `org.apache.druid.server.lookup.cache.LookupCoordinatorManager` is consuming 90% of the heap. (attaches some screenshots of the heap dump analysis)

## Expected Behavior
- `LookupCoordinatorManager` should **not duplicate** lookups unnecessarily for each process.
- Lookups should have a **shared or optimized memory footprint** across processes.
- The memory overhead for lookups should remain **proportional** to their actual size, not process count.
Contributor guide
Research direction
Start by reading server/src/main/java/org/apache/druid/server/lookup/cache/LookupCoordinatorManager.java at the 24.0.0 implementation, focusing on how knownOldState is maintained for processes. Use the reported heap-dump findings and lookup sizes to trace the duplication. Done means lookup memory no longer scales with the number of query and data-server processes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100