Kafka ingestion ignores segment granularities that aren't "simple" granularities
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
### Affected Version
0.20.0
### Description
This is the granularity spec being used in a kafka ingestion task
```
"granularitySpec": {
"type": "uniform",
"segmentGranularity": {"type": "period", "period": "PT3H"},
"queryGranularity": "NONE",
"rollup": false
}
```
The `taskDuration` is also set to 3H or more. But druid is creating segments for each hour. For example, this is one of the segment ids: `mydatasource_2021-04-28T20:00:00.000Z_2021-04-28T21:00:00.000Z_2021-04-28T20:03:38.441Z`. The max total rows or the max rows per segment limits are not being hit. Even if those limits were being hit, I'd still expect the segment to be marked for 3 hours and more partitions to get created as more data comes in for the same time period.
The same thing happens even with an equivalent `duration` granularity (10800000).
The documentation [states](https://druid.apache.org/docs/0.20.0/ingestion/index.html#granularityspec) that "Any granularity can be provided here." with a [link](https://druid.apache.org/docs/0.20.0/querying/granularities.html) to the query granularities page. So it seems like either this is expected behaviour and the documentation is incorrect or this is a bug and what the documentation says is the expected behaviour.
After unsuccessfully trying to get a response on an older issue #6170 (https://github.com/apache/druid/issues/6170#issuecomment-815946739, https://github.com/apache/druid/issues/6170#issuecomment-817947549, https://github.com/apache/druid/issues/6170#issuecomment-821286942) that seems to be the same bug and on the google group too, I am opening this new issue.
Contributor guide
Research direction
Start with the Kafka ingestion task and the GranularitySpec documentation linked in the report, then reproduce the PT3H and duration-granularity examples on Druid 0.20.0. Trace how the segmentGranularity value is handled and compare the produced segment intervals with the documented expectation. Done means the behavior is clarified or corrected for non-simple granularities, with the documentation matching the result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- databases, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100