apache / apache/airflow

DatabricksSubmitRunOperator fails when auto-injecting DAG params that contain None (nullable Param defaults)

Open
#71,776 0 comments 1 reaction 0 assignees View on GitHub
area:providers kind:bug needs-triage provider:databricks
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Under which category would you file this issue?

Providers

### Apache Airflow version

3.3.1

### What happened and how to reproduce it?

After upgrading `apache-airflow-providers-databricks` from 7.8.1 to 7.18.1, `DatabricksSubmitRunOperator` with a `python_wheel_task` that does **not** set `named_parameters` fails at submit time when the DAG defines nullable `Param`s (default `None`):

```
[2026-08-17 09:56:32] ERROR - Task failed with exception
AirflowException: Type used for parameter json[python_wheel_task][named_parameters][start_date_str] is not a number or a string
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 1579 in _run_task_and_map_outcome
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 197 in wrapper
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 2234 in _execute_task
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 197 in wrapper
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 2192 in _run_execute_callable
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/bases/operator.py", line 445 in wrapper
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/databricks/operators/databricks.py", line 871 in execute
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/bases/resumablejobmixin.py", line 200 in execute_resumable
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/databricks/operators/databricks.py", line 930 in submit_job
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/databricks/operators/databricks.py", line 907 in _prepare_submit_json
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/databricks/utils/databricks.py", line 50 in normalise_json_content
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/databricks/utils/databricks.py", line 50 in normalise_json_content
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/databricks/utils/databricks.py", line 50 in normalise_json_content
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/databricks/utils/databricks.py", line 53 in normalise_json_content
```

### How to reproduce
```python
from airflow.sdk import DAG, Param
from airflow.providers.databricks.operators.databricks import DatabricksSubmitRunOperator
with DAG(
dag_id="repro_databricks_none_params",
params={
"start_date_str": Param(default=None, type=["string", "null"]),
},
):
DatabricksSubmitRunOperator(
task_id="wheel_no_named_parameters",
new_cluster={...},
python_wheel_task={
"package_name": "my_package",
"entry_point": "my_entrypoint_with_no_args",
# named_parameters intentionally omitted
},
libraries=[...],
)
```

### What you think should happen instead?

If there are parameters with value None they should be simply omitted and not passed to a Databricks Run.

### Operating System

Debian 12

### Deployment

Official Apache Airflow Helm Chart

### Apache Airflow Provider(s)

databricks

### Versions of Apache Airflow Providers

```
apache-airflow-providers-amazon 9.34.0
apache-airflow-providers-cncf-kubernetes 10.21.0
apache-airflow-providers-common-compat 1.18.0
apache-airflow-providers-common-io 1.8.0
apache-airflow-providers-common-sql 2.1.0
apache-airflow-providers-databricks 7.18.1
apache-airflow-providers-fab 3.8.0
apache-airflow-providers-github 2.11.3
apache-airflow-providers-hashicorp 4.8.0
apache-airflow-providers-http 6.0.5
apache-airflow-providers-microsoft-mssql 4.7.0
apache-airflow-providers-mysql 6.6.1
apache-airflow-providers-postgres 7.0.1
apache-airflow-providers-sftp 6.0.1
apache-airflow-providers-slack 9.10.2
apache-airflow-providers-smtp 3.0.3
apache-airflow-providers-ssh 6.0.1
apache-airflow-providers-standard 1.17.0
```

### Official Helm Chart version

1.22.0 (latest released)

### Kubernetes Version

1.34.9

### Helm Chart configuration

_No response_

### Docker Image customizations

_No response_

### Anything else?

It happens every time it runs

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start with airflow/providers/databricks/operators/databricks.py, especially _prepare_submit_json and submit_job, then trace normalise_json_content in airflow/providers/databricks/utils/databricks.py. Reproduce the nullable Param case and add coverage for a python_wheel_task without named_parameters; done means None-valued parameters are omitted and the Databricks submission succeeds.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
75/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.