stackabletech / stackabletech/druid-operator
Add deploying ingestion specs from custom resources
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 12
- Forks
- 1
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 10
Description
Implementation ticket for #168
The Druid operator should be extended to be able to deploy ingestion specs from definitions provided in CRDs.
Ingestion specs can be defined by the user via a customresource, which is watched by a controller in the Druid operator that then provisions these specs.
The crd will contain at least the following:
- type of source (stream, flat file, ...)
- failure behavior
- none
- retry
- delete and retry
- specification of the ingest job
- inline
- reference to configmap
- reference to pvc (or file therein)
These objects will initially be considered read-only, so changes to them will not be propagated to Druid by the controller.
The initial implementation will not be a perfect ingestion task management solution, but rather a first attempt to offer something useful to our users.
The user needs to decide themselves what the appropriate failure behavior is for the spec they provide to the operator, whether duplicate data might be created by retrying etc.
The defined failure options should offer simple solutions for all scenarios:
- task is too complex -> none , the user will investigate themselves
- no duplicates expected -> retry , task is idempotent and can be retried
- duplicates possible -> delete and retry , not idempotent, to be on the safe side delete the target before retrying
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the related issue #168, then inspect the operator's CRD and controller entry points; no filenames or tests are identified here. Clarify how ingestion specs, ConfigMap/PVC references, and each failure behavior map to Druid before implementation. Done means the operator can provision the specified read-only CRD resources with the documented failure options.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, rust
- Domain
- data-engineering, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100