apache / apache/druid

Kafka ingestion ignores segment granularities that aren't "simple" granularities

Open
#11,179 5 comments 0 reactions 0 assignees View on GitHub
Area - Streaming Ingestion Bug
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Affected Version

0.20.0

### Description

This is the granularity spec being used in a kafka ingestion task
```
"granularitySpec": {
"type": "uniform",
"segmentGranularity": {"type": "period", "period": "PT3H"},
"queryGranularity": "NONE",
"rollup": false
}
```
The `taskDuration` is also set to 3H or more. But druid is creating segments for each hour. For example, this is one of the segment ids: `mydatasource_2021-04-28T20:00:00.000Z_2021-04-28T21:00:00.000Z_2021-04-28T20:03:38.441Z`. The max total rows or the max rows per segment limits are not being hit. Even if those limits were being hit, I'd still expect the segment to be marked for 3 hours and more partitions to get created as more data comes in for the same time period.
The same thing happens even with an equivalent `duration` granularity (10800000).

The documentation [states](https://druid.apache.org/docs/0.20.0/ingestion/index.html#granularityspec) that "Any granularity can be provided here." with a [link](https://druid.apache.org/docs/0.20.0/querying/granularities.html) to the query granularities page. So it seems like either this is expected behaviour and the documentation is incorrect or this is a bug and what the documentation says is the expected behaviour.

After unsuccessfully trying to get a response on an older issue #6170 (https://github.com/apache/druid/issues/6170#issuecomment-815946739, https://github.com/apache/druid/issues/6170#issuecomment-817947549, https://github.com/apache/druid/issues/6170#issuecomment-821286942) that seems to be the same bug and on the google group too, I am opening this new issue.

Contributor guide

Open the contributing guide

Research direction

Start with the Kafka ingestion task and the GranularitySpec documentation linked in the report, then reproduce the PT3H and duration-granularity examples on Druid 0.20.0. Trace how the segmentGranularity value is handled and compare the produced segment intervals with the documented expectation. Done means the behavior is clarified or corrected for non-simple granularities, with the documentation matching the result.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
databases, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.