apache / apache/druid

CompactSegments errors after upgrade to 24.0

Open
#13,132 3 comments 0 reactions 0 assignees View on GitHub
Area - Segment Balancing/Coordination Bug
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

Reported in Slack here: https://apachedruidworkspace.slack.com/archives/C0309C9L90D/p1663633728125109. Logs reproduced below.

I think what is happening here is that the changes in #12404 make the task `type` from the Overlord tasks API return `null` until the metadata store update finishes in the background. (See `SQLMetadataStorageActionHandler#populateTaskTypeAndGroupId`.) It happens automatically when the Overlord starts up and logs messages like:

> Populate fields task and group_id of task entry table [%s] from payload

And:

> Task migration for table [druid_tasks] successful

During this time, the Coordinator is unable to properly filter out non-compaction tasks from the task list. It tries to deserialize non-compaction tasks as ClientTaskQuery, which it can't do. (That class only supports a subset of task types.)

I believe this situation would resolve on its own once the metadata store update is complete.

Logs:

```
2022-09-19T23:08:35,342 ERROR [Coordinator-Exec--0] org.apache.druid.server.coordinator.DruidCoordinator - Caught exception, ignoring so that schedule keeps going.: {class=org.apache.druid.server.coordinator.DruidCoordinator, exceptionType=class java.lang.RuntimeException, exceptionMessage=com.fasterxml.jackson.databind.exc.InvalidTypeIdException: Please make sure to load all the necessary extensions and jars with type 'index_kafka' on 'druid/coordinator' service. Could not resolve type id 'index_kafka' as a subtype of `org.apache.druid.client.indexing.ClientTaskQuery` known type ids = [compact, kill] (for POJO property 'payload')
at [Source: (String)"{"task":"index_kafka_druid_v2_30a330c471e9073_afjlnglj","payload":{"type":"index_kafka","id":"index_kafka_druid_v2_30a330c471e9073_afjlnglj","resource":{"availabilityGroup":"index_kafka_druid_v2_30a330c471e9073","requiredCapacity":1},"dataSchema":{"dataSource":"druid_v2","timestampSpec":{"column":"timestamp","format":"millis","missingValue":null},"dimensionsSpec":{"dimensions":[{"type":"string","name":"companyID","multiValueHandling":"SORTED_ARRAY","createBitmapIndex":true},{"type":"string","nam"[truncated 5767 chars]; line: 1, column: 75] (through reference chain: org.apache.druid.client.indexing.TaskPayloadResponse["payload"])}
java.lang.RuntimeException: com.fasterxml.jackson.databind.exc.InvalidTypeIdException: Please make sure to load all the necessary extensions and jars with type 'index_kafka' on 'druid/coordinator' service. Could not resolve type id 'index_kafka' as a subtype of `org.apache.druid.client.indexing.ClientTaskQuery` known type ids = [compact, kill] (for POJO property 'payload')
at [Source: (String)"{"task":"index_kafka_druid_v2_30a330c471e9073_afjlnglj","payload":{"type":"index_kafka","id":"index_kafka_druid_v2_30a330c471e9073_afjlnglj","resource":{"availabilityGroup":"index_kafka_druid_v2_30a330c471e9073","requiredCapacity":1},"dataSchema":{"dataSource":"druid_v2","timestampSpec":{"column":"timestamp","format":"millis","missingValue":null},"dimensionsSpec":{"dimensions":[{"type":"string","name":"companyID","multiValueHandling":"SORTED_ARRAY","createBitmapIndex":true},{"type":"string","nam"[truncated 5767 chars]; line: 1, column: 75] (through reference chain: org.apache.druid.client.indexing.TaskPayloadResponse["payload"])
at org.apache.druid.client.indexing.HttpIndexingServiceClient.getTaskPayload(HttpIndexingServiceClient.java:351) ~[druid-server-24.0.0.jar:24.0.0]
at org.apache.druid.server.coordinator.duty.CompactSegments.run(CompactSegments.java:143) ~[druid-server-24.0.0.jar:24.0.0]
at org.apache.druid.server.coordinator.DruidCoordinator$DutiesRunnable.run(DruidCoordinator.java:948) ~[druid-server-24.0.0.jar:24.0.0]
at org.apache.druid.server.coordinator.DruidCoordinator$2.call(DruidCoordinator.java:721) ~[druid-server-24.0.0.jar:24.0.0]
at org.apache.druid.server.coordinator.DruidCoordinator$2.call(DruidCoordinator.java:714) ~[druid-server-24.0.0.jar:24.0.0]
at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$4.run(ScheduledExecutors.java:163) ~[druid-core-24.0.0.jar:24.0.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_275]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_275]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_275]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:1.8.0_275]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_275]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_275]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_275]
```

Contributor guide

Open the contributing guide

Research direction

Start with CompactSegments.java and HttpIndexingServiceClient.java, then trace how SQLMetadataStorageActionHandler#populateTaskTypeAndGroupId affects the Overlord tasks API during startup. Use the reproduced InvalidTypeIdException and the task-type filtering path to determine the expected behavior while migration is running; done means the Coordinator no longer fails on non-compaction tasks in that situation, with regression coverage if an existing test area is identified.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.