Cannot retrieve proper tasks list from Overlord Task API with createdTimeInterval paramater
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
### Affected Version
0.21.1
### Description
(docs)[https://druid.apache.org/docs/0.21.1/operations/api-reference.html#get-14]
From the documentation, `/druid/indexer/v1/tasks` API accept `createdTimeInterval` parameter.
If I call API without `createdTimeInterval` parameter, I can retrieve 1 task which createdTime is `2021-08-12T23:51:23.151Z`.
```sh
curl -X http://{OVERLORD_HOST}:8090/druid/indexer/v1/tasks?datasource=new-data-source
```
```json
[
{
"id": "index_parallel_new-data-source_mfchdpon_2021-08-12T23:51:23.142Z",
"groupId": "index_parallel_new-data-source_mfchdpon_2021-08-12T23:51:23.142Z",
"type": "index_parallel",
"createdTime": "2021-08-12T23:51:23.151Z",
"queueInsertionTime": "1970-01-01T00:00:00.000Z",
"statusCode": "SUCCESS",
"status": "SUCCESS",
"runnerStatusCode": "NONE",
"duration": 25546,
"location": {
"host": "{MIDDLE_MANAGER_HOST}",
"port": 8100,
"tlsPort": -1
},
"dataSource": "new-data-source",
"errorMsg": null
}
]
```
The createdTime of this task was `2021-08-12T23:51:23.151Z`, so `createdTimeInterval` "2021-08-12T23:50:00.000Z_2021-08-13T00:00:00.000Z" should contain this task. But I retrieve empty task.
```sh
curl -X http://{OVERLORD_HOST}:8090/druid/indexer/v1/tasks?datasource=new-data-source&createdTimeInterval=2021-08-12T23:50:00.000Z_2021-08-13T00:00:00.000Z
```
```json
[]
```
But when I specify `createdTimeInterval` "2021-01-01T00:00:00.000Z_2021-01-02T00:00:10.000Z", I can retrieve 1 task which createdTime was `2021-08-12T23:51:23.151Z`
```sh
curl -X http://{OVERLORD_HOST}:8090/druid/indexer/v1/tasks?datasource=new-data-source&createdTimeInterval=2021-01-01T00:00:00.000Z_2021-01-02T00:00:10.000Z
```
```json
[
{
"id": "index_parallel_new-data-source_mfchdpon_2021-08-12T23:51:23.142Z",
"groupId": "index_parallel_new-data-source_mfchdpon_2021-08-12T23:51:23.142Z",
"type": "index_parallel",
"createdTime": "2021-08-12T23:51:23.151Z",
"queueInsertionTime": "1970-01-01T00:00:00.000Z",
"statusCode": "SUCCESS",
"status": "SUCCESS",
"runnerStatusCode": "NONE",
"duration": 25546,
"location": {
"host": "{MIDDLE_MANAGER_HOST}",
"port": 8100,
"tlsPort": -1
},
"dataSource": "new-data-source",
"errorMsg": null
}
]
```
#### Search from code
When call task API with `createdTimeInterval` parameter, below code execute.
It calculate time duration from time interval.
https://github.com/apache/druid/blob/druid-0.21.1/indexing-service/src/main/java/org/apache/druid/indexing/overlord/http/OverlordResource.java#L616
```java
Duration createdTimeDuration = null;
if (createdTimeInterval != null) {
final Interval theInterval = Intervals.of(StringUtils.replace(createdTimeInterval, "_", "/"));
createdTimeDuration = theInterval.toDuration();
}
final List> taskInfoList =
taskStorageQueryAdapter.getCompletedTaskInfoByCreatedTimeDuration(maxCompletedTasks, createdTimeDuration, dataSource);
```
And `getCompletedTaskInfoByCreatedTimeDuration` call `getRecentlyCreatedAlreadyFinishedTaskInfo` method.
https://github.com/apache/druid/blob/druid-0.21.1/indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskStorageQueryAdapter.java#L61
```java
public List> getCompletedTaskInfoByCreatedTimeDuration(
@Nullable Integer maxTaskStatuses,
@Nullable Duration duration,
@Nullable String dataSource
)
{
return storage.getRecentlyCreatedAlreadyFinishedTaskInfo(maxTaskStatuses, duration, dataSource);
}
```
`getRecentlyCreatedAlreadyFinishedTaskInfo` method is coping completed task list which createdTime is (now - duration) ~ (now).
https://github.com/apache/druid/blob/druid-0.21.1/indexing-service/src/main/java/org/apache/druid/indexing/overlord/MetadataTaskStorage.java#L223
```java
@Override
public List> getRecentlyCreatedAlreadyFinishedTaskInfo(
@Nullable Integer maxTaskStatuses,
@Nullable Duration durationBeforeNow,
@Nullable String datasource
)
{
return ImmutableList.copyOf(
handler.getCompletedTaskInfo(
DateTimes.nowUtc()
.minus(durationBeforeNow == null ? config.getRecentlyFinishedThreshold() : durationBeforeNow),
maxTaskStatuses,
datasource
)
);
}
```
So I think create `getCompletedTaskInfoByCreatedTimeInterval` method from `TaskStorageQueryAdapter` class which arguments are `maxTaskStatuses`, `interval`, and `dataSource`.
Contributor guide
Research direction
Start in indexing-service/src/main/java/org/apache/druid/indexing/overlord/http/OverlordResource.java and follow TaskStorageQueryAdapter.java to MetadataTaskStorage.java. Compare the interval handling with getRecentlyCreatedAlreadyFinishedTaskInfo and verify the API against the reported curl cases. Done means createdTimeInterval returns only tasks within the requested interval, with tests covering the behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- api, backend-api-design
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100