apache / apache/druid

Historicals OOM Loop with Globally Cached JDBC Lookups

Open
#18,740 2 comments 0 reactions 0 assignees View on GitHub
stale Uncategorized problem report
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

The lookups we are fetching are 2.2G total with about 42 lookups. We have a 5 minute polling period with each lookup taking about 6 seconds while using 6 lookupThreads. Our heap has 50GB and it still goes OOM. So the lookups should complete well within the 5 minute polling period. However, the system sometimes does become unstable and is very sensitive on database performance. We realized the lookups were the causing OOM because when we split the lookups across multiple database replicas they recovered. Also after checking heap dump the lookups were taking most of the memory within a sample smaller heap of around 20G instead of analyzing a 50G heap. The pattern also noticed was that once a historical went into the high memory bad state where it would go OOM, it would continue going OOM over and over, and lookup connections would increase significantly.

### Affected Version

32.0.0

### Description

Please include as much detailed information about the problem as possible.
- 28 historicals, 3 broker/routers, 2 coordinators
- Configs attached
- Setup lookups with large 2.2G total with 42 lookups
- No error messages we just see the process keeps restarting trying to connect back to zookeeper and latencies jump significantly into the 10s of seconds to minutes
- Tried multiple changes with druid lookup threads increased decreased, num processing thread changes for historicals and various other configs, offHeap (got these errors and queries failing had to revert)

Offheap errors: ```2025-04-10T05:02:43,051 ERROR [Cleaner-0] org.apache.druid.server.lookup.namespace.cache.OffHeapNamespaceExtractionCacheManager - OffHeapNamespaceExtractionCacheManager.disposeCache() was not called, disposed resources by the JVM
2025-04-10T05:02:43,284 ERROR [Cleaner-0] org.apache.druid.server.lookup.namespace.cache.OffHeapNamespaceExtractionCacheManager - OffHeapNamespaceExtractionCacheManager.disposeCache() was not called, disposed resources by the JVM
2025-04-10T05:08:09,698 ERROR [Cleaner-0] org.apache.druid.server.lookup.namespace.cache.OffHeapNamespaceExtractionCacheManager - OffHeapNamespaceExtractionCacheManager.disposeCache() was not called, disposed resources by the JVM
2025-04-10T05:08:54,997 ERROR [Cleaner-0] org.apache.druid.server.lookup.namespace.cache.OffHeapNamespaceExtractionCacheManager - OffHeapNamespaceExtractionCacheManager.disposeCache() was not called, disposed resources by the JVM
2025-04-10T05:11:28,411 ERROR [Cleaner-0] org.apache.druid.server.lookup.namespace.cache.OffHeapNamespaceExtractionCacheManager - OffHeapNamespaceExtractionCacheManager.disposeCache() was not called, disposed resources by the JVM
2025-04-10T05:13:24,720 ERROR [Cleaner-0] org.apache.druid.server.lookup.namespace.cache.OffHeapNamespaceExtractionCacheManager - OffHeapNamespaceExtractionCacheManager.disposeCache() was not called, disposed resources by the JVM```

Contributor guide

Open the contributing guide

Research direction

Start with the attached configurations and the heap-dump observations for the 42 JDBC lookups totaling 2.2G. No source file, entry point, or test is named; investigate the globally cached lookup behavior and repeated connection growth, then reproduce the historical OOM loop and define a fix that keeps the process stable during polling.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.