ingestSegment firehose is not friendly to the sampler
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
### Affected Version
0.16, 0.15 (and technically all versions of Druid)
### Description
When sampling a datasource with the data loader sampler (via `ingestSegment` firehose) having intervals set a bit too large makes a http call that never returns in time because it loads way too much data and does not listen to the sampler `maxRows` or `timeoutMs` settings.
The root cause of it can be found in the abnormally large amount of work that is done in the ingestSegmentFirehose constructor which stalls everything and does not let the sampler do its thing.
Here is an example request:
```js
{
"type": "index",
"spec": {
"type": "index",
"ioConfig": {
"type": "index",
"firehose": {
"type": "ingestSegment",
"interval": "2010-06-27/2020-06-27", // Pray that this interval does not have many segments
"dataSource": "wikipedia"
}
},
"dataSchema": {
"dataSource": "sample",
"parser": {
"type": "string",
"parseSpec": {
"format": "regex",
"pattern": "(.*)",
"columns": [
"a"
],
"dimensionsSpec": {},
"timestampSpec": {
"column": "!!!_no_such_column_!!!",
"missingValue": "2010-01-01T00:00:00Z"
}
}
}
}
},
"samplerConfig": {
"numRows": 500,
"timeoutMs": 15000
}
}
```
### Impact:
Fixing this will make the data loader work nicer for segment re-ingest

Contributor guide
Research direction
Start at the ingestSegmentFirehose constructor, which the issue identifies as doing excessive work before the data loader sampler can act. Review how the ingestSegment firehose interacts with sampler maxRows and timeoutMs, using the provided request as the reproduction case. Done means segment re-ingest sampling no longer stalls on large intervals and respects those sampler settings.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100