stackabletech / stackabletech/druid-operator

Add deploying ingestion specs from custom resources

Open
#226 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
12
Forks
1
Avg merge
1d 17h
Merged PRs (30d)
10

Description

Implementation ticket for #168

The Druid operator should be extended to be able to deploy ingestion specs from definitions provided in CRDs.

Ingestion specs can be defined by the user via a customresource, which is watched by a controller in the Druid operator that then provisions these specs.

The crd will contain at least the following:

  • type of source (stream, flat file, ...)
  • failure behavior
    • none
    • retry
    • delete and retry
  • specification of the ingest job
    • inline
    • reference to configmap
    • reference to pvc (or file therein)

These objects will initially be considered read-only, so changes to them will not be propagated to Druid by the controller.

The initial implementation will not be a perfect ingestion task management solution, but rather a first attempt to offer something useful to our users.
The user needs to decide themselves what the appropriate failure behavior is for the spec they provide to the operator, whether duplicate data might be created by retrying etc.
The defined failure options should offer simple solutions for all scenarios:

  • task is too complex -> none , the user will investigate themselves
  • no duplicates expected -> retry , task is idempotent and can be retried
  • duplicates possible -> delete and retry , not idempotent, to be on the safe side delete the target before retrying

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the related issue #168, then inspect the operator's CRD and controller entry points; no filenames or tests are identified here. Clarify how ingestion specs, ConfigMap/PVC references, and each failure behavior map to Druid before implementation. Done means the operator can provision the specified read-only CRD resources with the documented failure options.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, rust
Domain
data-engineering, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.