apache / apache/druid

ingestSegment firehose is not friendly to the sampler

Open
#8,448 0 comments 0 reactions 0 assignees View on GitHub
Area - Batch Ingestion Bug
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Affected Version

0.16, 0.15 (and technically all versions of Druid)

### Description

When sampling a datasource with the data loader sampler (via `ingestSegment` firehose) having intervals set a bit too large makes a http call that never returns in time because it loads way too much data and does not listen to the sampler `maxRows` or `timeoutMs` settings.

The root cause of it can be found in the abnormally large amount of work that is done in the ingestSegmentFirehose constructor which stalls everything and does not let the sampler do its thing.

Here is an example request:

```js
{
"type": "index",
"spec": {
"type": "index",
"ioConfig": {
"type": "index",
"firehose": {
"type": "ingestSegment",
"interval": "2010-06-27/2020-06-27", // Pray that this interval does not have many segments
"dataSource": "wikipedia"
}
},
"dataSchema": {
"dataSource": "sample",
"parser": {
"type": "string",
"parseSpec": {
"format": "regex",
"pattern": "(.*)",
"columns": [
"a"
],
"dimensionsSpec": {},
"timestampSpec": {
"column": "!!!_no_such_column_!!!",
"missingValue": "2010-01-01T00:00:00Z"
}
}
}
}
},
"samplerConfig": {
"numRows": 500,
"timeoutMs": 15000
}
}
```

### Impact:

Fixing this will make the data loader work nicer for segment re-ingest

![image](https://user-images.githubusercontent.com/177816/64070571-31e6c700-cc18-11e9-8225-b6c03063b6fa.png)

Contributor guide

Open the contributing guide

Research direction

Start at the ingestSegmentFirehose constructor, which the issue identifies as doing excessive work before the data loader sampler can act. Review how the ingestSegment firehose interacts with sampler maxRows and timeoutMs, using the provided request as the reproduction case. Done means segment re-ingest sampling no longer stalls on large intervals and respects those sampler settings.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.