job table missing `dag_id` and `executor_class` for BackfillJob type
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
### Apache Airflow version
2.9.1
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
When using the `airflow dags backfill` command to execute a backfill for any DAG, I noticed that the `dag_id` and `executor_class` columns in the job table are empty(null).
Results in entries in the job table like this:
```
124460, NULL, running, BackfillJob, 2024-06-05 01:03:40.936253+00, NULL, 2024-06-05 01:22:41.634367+00, NULL, hoge-5566755f9-wbw57, airflow
```
However, in versions before 2.6.0, the entries in the job table included the dag_id and executor_class as expected:
```
2, bash_operator, running, BackfillJob, 2024-06-05 02:25:32.905082+00, NULL, 2024-06-05 02:25:48.017582+00, CeleryExecutor, a5dec317208c, airflow
```
I'm not sure exactly where the issue lies (or if this is a feature rather than a bug), but I will take a closer look at the code later. If I find the problem, I might submit a PR with a fix.
### What you think should happen instead?
I'm not sure, but it looks like this PR https://github.com/apache/airflow/pull/30376 changed a lot in BackfillJobRunner, Maybe there is something we missed in there. Additionally, these changes are not even mentioned in the `RELEASE_NOTES.rst`.
> reviewable stages: https://github.com/apache/airflow/pull/30255 -> https://github.com/apache/airflow/pull/30302 -> https://github.com/apache/airflow/pull/30308 -> this PR.
### How to reproduce
1. any simple dag file
2. run airflow dags backfill with above dag's id
3. check webserver's Browse > Jobs, and limit the Job Type to BackfillJob. or just check record in job table when JobType = 'BackfillJob'
This issue can be reproduced in versions from 2.6.0 to the latest version.
### Operating System
MacOS 14.4.1
### Versions of Apache Airflow Providers
None
### Deployment
Docker-Compose
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start with BackfillJobRunner and compare the changes in PRs 30376, 30255, 30302, and 30308. Reproduce the issue with a simple DAG and an airflow dags backfill run, then inspect the resulting BackfillJob row. Done means dag_id and executor_class are populated as expected in the job table for supported Airflow versions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100