stackabletech / stackabletech/airflow-operator

Git-sync with kubernetesExecutors and using https

Open
#468 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

type/bug
Dominant language
Rust
Stars
32
Forks
6
Avg merge
1d 7h
Merged PRs (30d)
13

Description

Affected Stackable version

Nightly

Affected Apache Airflow version

2.8.1

Current and expected behavior

Using this AirflowCluster:

---
apiVersion: airflow.stackable.tech/v1alpha1
kind: AirflowCluster
metadata:
  name: airflow
spec:
  image:
    productVersion: 2.8.1
  clusterConfig:
    loadExamples: false
    exposeConfig: false
    listenerClass: external-unstable
    credentialsSecret: simple-airflow-credentials
    dagsGitSync: 
    - repo: https://custom.gitlab.com/repos/airflow-dags.git
      branch: "main" 
      gitFolder: "dags" 
      depth: 10 
      wait: 20 
      credentialsSecret: gitcredentials
      gitSyncConf: 
        --rev: HEAD 
        --git-config: http.sslCAInfo:/tmp/ca-cert/ca.crt
    volumeMounts:
    - name: "cert-ca"
      mountPath: "/tmp/ca-cert"  
    volumes:
    - name: "cert-ca"
      configMap:
        name: "cert-ca"
        items: 
        - key: "ca.crt"
          path: "ca.crt"
  webservers:
    roleGroups:
      default:
        replicas: 1
    podOverrides:
      spec:
        containers:
          - name: gitsync-1
            volumeMounts:
            - name: "cert-ca"
              mountPath: "/tmp/ca-cert"  
  kubernetesExecutors:
    config:
      resources:
        cpu:
          min: 400m
          max: 800m
        memory:
          limit: 2Gi
    podOverrides:
      spec:
        containers:
          - name: gitsync-1
            volumeMounts:
              - name: "cert-ca"
                mountPath: "/tmp/ca-cert"
  schedulers:
    roleGroups:
      default:
        replicas: 1
    podOverrides:
      spec:
        containers:
          - name: gitsync-1
            volumeMounts:
            - name: "cert-ca"
              mountPath: "/tmp/ca-cert"

The gitSyncConf does not seem to get propagated for the kubernetesExecutors. Also the git-sync container has a one-time=true flag set for the git-sync command

Follow up this MR: https://github.com/stackabletech/airflow-operator/pull/456#issuecomment-2230116710

Expected behavior: the webserver, scheduler and kubernetesExecutors are configured exactly the same for the git-sync containers

Possible solution

No response

Additional context

No response

Environment

No response

Would you like to work on fixing this bug?

None

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Compare the git-sync settings shown for the webserver and scheduler with those generated for kubernetesExecutors, focusing on propagation of gitSyncConf and the one-time flag. Review the discussion in PR #456, especially the linked issue comment, then verify that all three workload types produce equivalent git-sync configuration for HTTPS repositories.

Written by the indexing model from the issue text.

Assessment

Tech stack
git, kubernetes, rust
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.