apache / apache/druid

Ingest query returns SUCCESS before generated segments are loaded even when waitUntilSegmentsLoad is true

Open
#19,003 0 comments 0 reactions 0 assignees View on GitHub
Uncategorized problem report
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Affected Version

35.0.1

### Description

When running an MSQ ingestion with the `waitUntilSegmentsLoad` option enabled in the query context, the task may return a `SUCCESS` status even though the generated segments are not yet loaded.

This behavior is **intermittent** and does not occur systematically for every ingestion.

When the issue occurs, any query against the target datasource fails with the following error:
```
{
"error": "druidException",
"errorCode": "invalidInput",
"persona": "USER",
"category": "INVALID_INPUT",
"errorMessage": "Object 'my-datasource' not found (line [1], column [753])",
"context": {
"sourceType": "sql",
"line": "1",
"column": "753",
"endLine": "1",
"endColumn": "790"
}
}
```
Example query (columns removed for confidentiality):
```
SELECT __time,other_column FROM "my-datasource" LIMIT 1
```
Anonymized SQL query:
```
INSERT INTO "my-datasource"
WITH ext AS (
SELECT *
FROM TABLE(
EXTERN(
'{"type":"hdfs","paths":"hdfs://hdfs-namenodes:8020/data.parquet"}',
'{"type":"parquet"}'
)
)
EXTEND ( start_date BIGINT, other_column VARCHAR )
)
SELECT MILLIS_TO_TIMESTAMP(start_date) AS __time, other_column
FROM ext
PARTITIONED BY ALL
```
Query context:
```
{
"__user": "druid_system",
"finalize": true,
"maxNumTasks": 6,
"maxParseExceptions": 0,
"queryId": "d7120b41-b2a4-4541-9723-655824523f29",
"rowBasedFrameType": 19,
"sqlInsertSegmentGranularity": "{\"type\":\"all\"}",
"sqlQueryId": "d7120b41-b2a4-4541-9723-655824523f29",
"startTime": "2026-02-10T11:20:29.592Z",
"taskAssignment": "auto",
"waitUntilSegmentsLoad": true,
"windowFunctionOperatorTransformation": true
}
```

A similar issue can also occur where the task returns a `SUCCESS` status while not all columns of the datasource are available yet. In this case, the datasource exists and is queryable, but some columns are temporarily missing.

More informations:
- Cluster setup:
- 2 brokers
- 1 coordinator
- 2 historicals
- Deployment:
- Middle manager less mode
- Kubernetes

Contributor guide

Open the contributing guide

Research direction

Start with the MSQ ingestion lifecycle and the waitUntilSegmentsLoad query-context option, using the provided INSERT query and deployment details to reproduce the intermittent result. Check the task's SUCCESS transition against datasource availability and schema visibility. Done means SUCCESS is returned only after all generated segments and columns are available for queries.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, java, kubernetes
Domain
data-engineering, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.