apache / apache/airflow

Chart scheduler persistence should not depend on celery configuration

Open
#70,784 2 comments 0 reactions 0 assignees View on GitHub
area:helm-chart kind:bug
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Under which category would you file this issue?

Helm chart

### Apache Airflow version

main

### What happened and how to reproduce it?

I noticed what the logic to determine if a scheduler is a deployment or a statefulset now looks at the `.Values.workers.celery.persistence.enabled `. This used to be the more general `.Values.workers.persistence.enabled`, but now this coupling does not make sense.

This was a miss when celery was split out of the general workers section.

### What you think should happen instead?

I haven't spent much time thinking about it, but maybe we just need a new `scheduler.persistence` to control it? Or `workers.local.persistence`?

Whoever picks this up should explore the possible options.

### Operating System

_No response_

### Deployment

Official Apache Airflow Helm Chart

### Apache Airflow Provider(s)

_No response_

### Versions of Apache Airflow Providers

_No response_

### Official Helm Chart version

main (development)

### Kubernetes Version

_No response_

### Helm Chart configuration

_No response_

### Docker Image customizations

_No response_

### Anything else?

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Trace the Helm chart logic that decides whether the scheduler is a Deployment or StatefulSet, starting with its use of `.Values.workers.celery.persistence.enabled`. Compare that coupling with the former `.Values.workers.persistence.enabled`, then evaluate where an independent scheduler persistence setting should live; done means scheduler persistence no longer depends on Celery configuration and the selected behavior is consistent.

Written by the indexing model from the issue text.

Assessment

Tech stack
helm, kubernetes
Domain
devops, infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.