stackabletech / stackabletech/airflow-operator

Fix: Remove the possibility to add multiple git-sync resources as DAG backends

Open
#760 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

good first issue type/feature-improvement
Dominant language
Rust
Stars
32
Forks
6
Avg merge
1d 7h
Merged PRs (30d)
13

Description

Context

We have tried to implement multiple git-sync resources into one DAGS_CORE_FOLDER which was successful.

However when the feature was completed we noticed problems with this approach like the local dag path bleeds into DAG logic. After investigating we found reasons to rather not do it: https://github.com/stackabletech/airflow-operator/issues/721#issuecomment-4038282638

This now leads to the decision to not expose a possibility to add multiple dags in the Airflow CR.

What needs to happen

We want to remove the possibility to

  clusterConfig:
    credentialsSecret: test-airflow-credentials
    dagsGitSync:
      - repo: https://github.com/a-pretty-repo
        branch: feat/beautiful-branch
        gitFolder: "a-passably-folder"
      - repo: https://github.com/a-grave-repo
        credentialsSecret: git-credentials
        gitSyncConf:
          # supply some config to check that safe.directory is correctly set
          --git-config: http.sslVerify:false
        gitFolder: "a-well-formed-folder"

and rather have something like:

  clusterConfig:
    credentialsSecret: test-airflow-credentials
    dagsGitSync:
      repo: https://github.com/the-master-repo
        branch: feat/mighty-branch
        gitFolder: "all-my-magic"

as leaving this option open suggest a incomplete feature and might raise expectation on us to implement it.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No source files or tests are named in the issue. Start by locating the Airflow CR definition and all references to clusterConfig.dagsGitSync; done means the configuration accepts one git-sync resource rather than a list and the existing behavior remains covered by the project's checks.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, rust
Domain
backend, infrastructure
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.