DatabricksSubmitRunOperator fails when auto-injecting DAG params that contain None (nullable Param defaults)
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
### Under which category would you file this issue?
Providers
### Apache Airflow version
3.3.1
### What happened and how to reproduce it?
After upgrading `apache-airflow-providers-databricks` from 7.8.1 to 7.18.1, `DatabricksSubmitRunOperator` with a `python_wheel_task` that does **not** set `named_parameters` fails at submit time when the DAG defines nullable `Param`s (default `None`):
```
[2026-08-17 09:56:32] ERROR - Task failed with exception
AirflowException: Type used for parameter json[python_wheel_task][named_parameters][start_date_str] is not a number or a string
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 1579 in _run_task_and_map_outcome
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 197 in wrapper
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 2234 in _execute_task
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 197 in wrapper
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 2192 in _run_execute_callable
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/bases/operator.py", line 445 in wrapper
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/databricks/operators/databricks.py", line 871 in execute
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/bases/resumablejobmixin.py", line 200 in execute_resumable
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/databricks/operators/databricks.py", line 930 in submit_job
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/databricks/operators/databricks.py", line 907 in _prepare_submit_json
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/databricks/utils/databricks.py", line 50 in normalise_json_content
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/databricks/utils/databricks.py", line 50 in normalise_json_content
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/databricks/utils/databricks.py", line 50 in normalise_json_content
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/databricks/utils/databricks.py", line 53 in normalise_json_content
```
### How to reproduce
```python
from airflow.sdk import DAG, Param
from airflow.providers.databricks.operators.databricks import DatabricksSubmitRunOperator
with DAG(
dag_id="repro_databricks_none_params",
params={
"start_date_str": Param(default=None, type=["string", "null"]),
},
):
DatabricksSubmitRunOperator(
task_id="wheel_no_named_parameters",
new_cluster={...},
python_wheel_task={
"package_name": "my_package",
"entry_point": "my_entrypoint_with_no_args",
# named_parameters intentionally omitted
},
libraries=[...],
)
```
### What you think should happen instead?
If there are parameters with value None they should be simply omitted and not passed to a Databricks Run.
### Operating System
Debian 12
### Deployment
Official Apache Airflow Helm Chart
### Apache Airflow Provider(s)
databricks
### Versions of Apache Airflow Providers
```
apache-airflow-providers-amazon 9.34.0
apache-airflow-providers-cncf-kubernetes 10.21.0
apache-airflow-providers-common-compat 1.18.0
apache-airflow-providers-common-io 1.8.0
apache-airflow-providers-common-sql 2.1.0
apache-airflow-providers-databricks 7.18.1
apache-airflow-providers-fab 3.8.0
apache-airflow-providers-github 2.11.3
apache-airflow-providers-hashicorp 4.8.0
apache-airflow-providers-http 6.0.5
apache-airflow-providers-microsoft-mssql 4.7.0
apache-airflow-providers-mysql 6.6.1
apache-airflow-providers-postgres 7.0.1
apache-airflow-providers-sftp 6.0.1
apache-airflow-providers-slack 9.10.2
apache-airflow-providers-smtp 3.0.3
apache-airflow-providers-ssh 6.0.1
apache-airflow-providers-standard 1.17.0
```
### Official Helm Chart version
1.22.0 (latest released)
### Kubernetes Version
1.34.9
### Helm Chart configuration
_No response_
### Docker Image customizations
_No response_
### Anything else?
It happens every time it runs
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start with airflow/providers/databricks/operators/databricks.py, especially _prepare_submit_json and submit_job, then trace normalise_json_content in airflow/providers/databricks/utils/databricks.py. Reproduce the nullable Param case and add coverage for a python_wheel_task without named_parameters; done means None-valued parameters are omitted and the Databricks submission succeeds.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 75/100