apache / apache/druid

NumShards will cause broker can't load segments

Open
#8,676 2 comments 0 reactions 0 assignees View on GitHub
Area - Querying Bug
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

NumShards will cause broker can't load segments

### Affected Version

0.9.2/0.12.3

### Description

- We load batch data which set `NumShards` to 100
- The number of batch data is 10, and their cardinality is 10 too.
- So although the reduce number is 100, but finally the shards will only 10 ,and their shards number is not continuous.
- Broker will add them in `incompletePartitionsTimeline` when creating versionedIntervalTimeline.
- The query about this datasource will return `null`

Contributor guide

Open the contributing guide

Research direction

Reproduce the case using batch data with NumShards set to 100, 10 batches, and cardinality 10. Trace how the broker adds the resulting non-contiguous shards to incompletePartitionsTimeline while creating versionedIntervalTimeline. Done means queries for the datasource return results instead of null.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.