apache / apache/druid

Do not Load lookups on tasks that will not use them

Open
#13,324 2 comments 0 reactions 0 assignees View on GitHub
Area - Lookups Feature/Change Description
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Description

At the moment, all druid tasks types attempt to load lookups, even when not needed:

1. Some task types, have absolutely no need to do this, such as compaction, or kill tasks
2. native batch ingest: only load lookups that are actually needed by transformSpec or metricsSpec
3. MSQ: only load lookups that are actually needed by the query

### Motivation

Lookups can be large. and should not consume resources on tasks that have no use for them. It's a waste of resources.

Contributor guide

Open the contributing guide

Research direction

Trace how lookup loading is initiated for Druid task types, then compare compaction, kill, native batch, and MSQ execution paths. Done means tasks that do not use lookups avoid loading them, while native batch and MSQ load only lookups required by their specifications or query.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.