apache / apache/druid

Cannot retrieve proper tasks list from Overlord Task API with createdTimeInterval paramater

Open
#11,589 1 comment 0 reactions 0 assignees View on GitHub
Bug
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Affected Version

0.21.1

### Description

(docs)[https://druid.apache.org/docs/0.21.1/operations/api-reference.html#get-14]

From the documentation, `/druid/indexer/v1/tasks` API accept `createdTimeInterval` parameter.

If I call API without `createdTimeInterval` parameter, I can retrieve 1 task which createdTime is `2021-08-12T23:51:23.151Z`.
```sh
curl -X http://{OVERLORD_HOST}:8090/druid/indexer/v1/tasks?datasource=new-data-source
```
```json
[
{
"id": "index_parallel_new-data-source_mfchdpon_2021-08-12T23:51:23.142Z",
"groupId": "index_parallel_new-data-source_mfchdpon_2021-08-12T23:51:23.142Z",
"type": "index_parallel",
"createdTime": "2021-08-12T23:51:23.151Z",
"queueInsertionTime": "1970-01-01T00:00:00.000Z",
"statusCode": "SUCCESS",
"status": "SUCCESS",
"runnerStatusCode": "NONE",
"duration": 25546,
"location": {
"host": "{MIDDLE_MANAGER_HOST}",
"port": 8100,
"tlsPort": -1
},
"dataSource": "new-data-source",
"errorMsg": null
}
]
```

The createdTime of this task was `2021-08-12T23:51:23.151Z`, so `createdTimeInterval` "2021-08-12T23:50:00.000Z_2021-08-13T00:00:00.000Z" should contain this task. But I retrieve empty task.
```sh
curl -X http://{OVERLORD_HOST}:8090/druid/indexer/v1/tasks?datasource=new-data-source&createdTimeInterval=2021-08-12T23:50:00.000Z_2021-08-13T00:00:00.000Z
```
```json
[]
```

But when I specify `createdTimeInterval` "2021-01-01T00:00:00.000Z_2021-01-02T00:00:10.000Z", I can retrieve 1 task which createdTime was `2021-08-12T23:51:23.151Z`
```sh
curl -X http://{OVERLORD_HOST}:8090/druid/indexer/v1/tasks?datasource=new-data-source&createdTimeInterval=2021-01-01T00:00:00.000Z_2021-01-02T00:00:10.000Z
```
```json
[
{
"id": "index_parallel_new-data-source_mfchdpon_2021-08-12T23:51:23.142Z",
"groupId": "index_parallel_new-data-source_mfchdpon_2021-08-12T23:51:23.142Z",
"type": "index_parallel",
"createdTime": "2021-08-12T23:51:23.151Z",
"queueInsertionTime": "1970-01-01T00:00:00.000Z",
"statusCode": "SUCCESS",
"status": "SUCCESS",
"runnerStatusCode": "NONE",
"duration": 25546,
"location": {
"host": "{MIDDLE_MANAGER_HOST}",
"port": 8100,
"tlsPort": -1
},
"dataSource": "new-data-source",
"errorMsg": null
}
]
```

#### Search from code
When call task API with `createdTimeInterval` parameter, below code execute.
It calculate time duration from time interval.
https://github.com/apache/druid/blob/druid-0.21.1/indexing-service/src/main/java/org/apache/druid/indexing/overlord/http/OverlordResource.java#L616
```java
Duration createdTimeDuration = null;
if (createdTimeInterval != null) {
final Interval theInterval = Intervals.of(StringUtils.replace(createdTimeInterval, "_", "/"));
createdTimeDuration = theInterval.toDuration();
}
final List> taskInfoList =
taskStorageQueryAdapter.getCompletedTaskInfoByCreatedTimeDuration(maxCompletedTasks, createdTimeDuration, dataSource);
```

And `getCompletedTaskInfoByCreatedTimeDuration` call `getRecentlyCreatedAlreadyFinishedTaskInfo` method.

https://github.com/apache/druid/blob/druid-0.21.1/indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskStorageQueryAdapter.java#L61
```java
public List> getCompletedTaskInfoByCreatedTimeDuration(
@Nullable Integer maxTaskStatuses,
@Nullable Duration duration,
@Nullable String dataSource
)
{
return storage.getRecentlyCreatedAlreadyFinishedTaskInfo(maxTaskStatuses, duration, dataSource);
}
```

`getRecentlyCreatedAlreadyFinishedTaskInfo` method is coping completed task list which createdTime is (now - duration) ~ (now).
https://github.com/apache/druid/blob/druid-0.21.1/indexing-service/src/main/java/org/apache/druid/indexing/overlord/MetadataTaskStorage.java#L223

```java
@Override
public List> getRecentlyCreatedAlreadyFinishedTaskInfo(
@Nullable Integer maxTaskStatuses,
@Nullable Duration durationBeforeNow,
@Nullable String datasource
)
{
return ImmutableList.copyOf(
handler.getCompletedTaskInfo(
DateTimes.nowUtc()
.minus(durationBeforeNow == null ? config.getRecentlyFinishedThreshold() : durationBeforeNow),
maxTaskStatuses,
datasource
)
);
}
```

So I think create `getCompletedTaskInfoByCreatedTimeInterval` method from `TaskStorageQueryAdapter` class which arguments are `maxTaskStatuses`, `interval`, and `dataSource`.

Contributor guide

Open the contributing guide

Research direction

Start in indexing-service/src/main/java/org/apache/druid/indexing/overlord/http/OverlordResource.java and follow TaskStorageQueryAdapter.java to MetadataTaskStorage.java. Compare the interval handling with getRecentlyCreatedAlreadyFinishedTaskInfo and verify the API against the reported curl cases. Done means createdTimeInterval returns only tasks within the requested interval, with tests covering the behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api, backend-api-design
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.