"assumeGrouped" behaves differently in native batch and hadoop tasks
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
### Affected Version
0.17.0 and master branches
### Description
`assumeGrouped` is a property supported with the single-dimension based range partitioning. The property was added for Hadoop ingestion first which is to accelerate the ingestion speed by skipping the first job to determine partitions when the input data is already partitioned.
The single-dimension based range partitioning was added in #8769 for parallel task, but the parallel task runs in 3 phases no matter what `assumeGrouped` is. Instead, if it's set, the task uses `PassthroughRowDimensionValueFilter` which assumes the input rows are unique.
I think this is a bug since the behavior of the property is supposed to be same in both Hadoop and native parallel ingestion, but don't think it's a release blocker for 0.17.0
Contributor guide
Research direction
Compare the native parallel ingestion and Hadoop ingestion paths that implement the single-dimension range-partitioning assumeGrouped property. Trace how the property affects task phases and row filtering, then add coverage showing equivalent behavior in both paths; done means native batch ingestion honors assumeGrouped consistently with Hadoop ingestion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100