apache / apache/druid

Fall Back To Known-Good Supervisor Spec On Invalid Spec

Open
#18,827 0 comments 0 reactions 0 assignees View on GitHub
Feature/Change Description
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Description
Currently (at least in Druid 28, I'm unsure if it's changed since,) when an invalid ingestion spec is received, the supervisor is deleted. This can cause a bug in ingestion spec submission/generation to quickly turn into a considerable outage. Instead, if an invalid supervisor spec is submitted then Druid should keep the supervisor around using the previous config and log the error.

### Motivation
I accidentally caused an overnight outage with our Druid ingestion due to a bug in supervisor config generation. It submitted a config which did not pass validation (taskCountMin was over taskCountMax) which Druid caught, but deleted the supervisor because it had no valid configuration. Our alerting didn't catch it because the entire supervisor was deleted, so there was no data reporting that it was down. This outage would've been prevented if Druid didn't destroy supervisors on bad config updates.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the supervisor spec validation and deletion path, using the taskCountMin-over-taskCountMax example to reproduce the failure. Done means an invalid submitted spec preserves the previous supervisor configuration and logs the validation error instead of deleting the supervisor.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, data-engineering, stream-processing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.