apache / apache/airflow

job table missing `dag_id` and `executor_class` for BackfillJob type

Open
#40,055 9 comments 0 reactions 0 assignees View on GitHub
area:backfill good first issue kind:bug
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Apache Airflow version

2.9.1

### If "Other Airflow 2 version" selected, which one?

_No response_

### What happened?

When using the `airflow dags backfill` command to execute a backfill for any DAG, I noticed that the `dag_id` and `executor_class` columns in the job table are empty(null).

Results in entries in the job table like this:
```
124460, NULL, running, BackfillJob, 2024-06-05 01:03:40.936253+00, NULL, 2024-06-05 01:22:41.634367+00, NULL, hoge-5566755f9-wbw57, airflow
```

However, in versions before 2.6.0, the entries in the job table included the dag_id and executor_class as expected:

```
2, bash_operator, running, BackfillJob, 2024-06-05 02:25:32.905082+00, NULL, 2024-06-05 02:25:48.017582+00, CeleryExecutor, a5dec317208c, airflow
```

I'm not sure exactly where the issue lies (or if this is a feature rather than a bug), but I will take a closer look at the code later. If I find the problem, I might submit a PR with a fix.

### What you think should happen instead?

I'm not sure, but it looks like this PR https://github.com/apache/airflow/pull/30376 changed a lot in BackfillJobRunner, Maybe there is something we missed in there. Additionally, these changes are not even mentioned in the `RELEASE_NOTES.rst`.

> reviewable stages: https://github.com/apache/airflow/pull/30255 -> https://github.com/apache/airflow/pull/30302 -> https://github.com/apache/airflow/pull/30308 -> this PR.

### How to reproduce

1. any simple dag file
2. run airflow dags backfill with above dag's id
3. check webserver's Browse > Jobs, and limit the Job Type to BackfillJob. or just check record in job table when JobType = 'BackfillJob'

This issue can be reproduced in versions from 2.6.0 to the latest version.

### Operating System

MacOS 14.4.1

### Versions of Apache Airflow Providers

None

### Deployment

Docker-Compose

### Deployment details

_No response_

### Anything else?

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start with BackfillJobRunner and compare the changes in PRs 30376, 30255, 30302, and 30308. Reproduce the issue with a simple DAG and an airflow dags backfill run, then inspect the resulting BackfillJob row. Done means dag_id and executor_class are populated as expected in the job table for supported Airflow versions.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.