Fall Back To Known-Good Supervisor Spec On Invalid Spec
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
### Description
Currently (at least in Druid 28, I'm unsure if it's changed since,) when an invalid ingestion spec is received, the supervisor is deleted. This can cause a bug in ingestion spec submission/generation to quickly turn into a considerable outage. Instead, if an invalid supervisor spec is submitted then Druid should keep the supervisor around using the previous config and log the error.
### Motivation
I accidentally caused an overnight outage with our Druid ingestion due to a bug in supervisor config generation. It submitted a config which did not pass validation (taskCountMin was over taskCountMax) which Druid caught, but deleted the supervisor because it had no valid configuration. Our alerting didn't catch it because the entire supervisor was deleted, so there was no data reporting that it was down. This outage would've been prevented if Druid didn't destroy supervisors on bad config updates.
Contributor guide
Research direction
Start by tracing the supervisor spec validation and deletion path, using the taskCountMin-over-taskCountMax example to reproduce the failure. Done means an invalid submitted spec preserves the previous supervisor configuration and logs the validation error instead of deleting the supervisor.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100