Do not Load lookups on tasks that will not use them
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
### Description
At the moment, all druid tasks types attempt to load lookups, even when not needed:
1. Some task types, have absolutely no need to do this, such as compaction, or kill tasks
2. native batch ingest: only load lookups that are actually needed by transformSpec or metricsSpec
3. MSQ: only load lookups that are actually needed by the query
### Motivation
Lookups can be large. and should not consume resources on tasks that have no use for them. It's a waste of resources.
Contributor guide
Research direction
Trace how lookup loading is initiated for Druid task types, then compare compaction, kill, native batch, and MSQ execution paths. Done means tasks that do not use lookups avoid loading them, while native batch and MSQ load only lookups required by their specifications or query.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100