apache / apache/druid

"assumeGrouped" behaves differently in native batch and hadoop tasks

Open
#9,168 6 comments 0 reactions 0 assignees View on GitHub
Area - Batch Ingestion Bug
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Affected Version

0.17.0 and master branches

### Description

`assumeGrouped` is a property supported with the single-dimension based range partitioning. The property was added for Hadoop ingestion first which is to accelerate the ingestion speed by skipping the first job to determine partitions when the input data is already partitioned.

The single-dimension based range partitioning was added in #8769 for parallel task, but the parallel task runs in 3 phases no matter what `assumeGrouped` is. Instead, if it's set, the task uses `PassthroughRowDimensionValueFilter` which assumes the input rows are unique.

I think this is a bug since the behavior of the property is supposed to be same in both Hadoop and native parallel ingestion, but don't think it's a release blocker for 0.17.0

Contributor guide

Open the contributing guide

Research direction

Compare the native parallel ingestion and Hadoop ingestion paths that implement the single-dimension range-partitioning assumeGrouped property. Trace how the property affects task phases and row filtering, then add coverage showing equivalent behavior in both paths; done means native batch ingestion honors assumeGrouped consistently with Hadoop ingestion.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.