Smarter selection of sdist packages to build
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
Currently we have an edge case that when we release a new version of provider that other providers depend on, the sdist build might have a problem with fiding the dependent packages in some PRS - when the PRs are modifying only some packages as they are not locally built.
When we build sdist providers, we only build them in some PRs for only affected providers in this PR - and their dependencies. The sdist provider builds are spit into chunks, and the chunks are built independently in parallel (for speed) which means that in some cases the required new version of package will be missing.
For example this happened in https://github.com/apache/airflow/pull/44018 (https://github.com/apache/airflow/actions/runs/11835585982) when modified standard provider,
```
× No solution found when resolving dependencies:
╰─▶ Because only apache-airflow-providers-common-sql<=1.19.0 is available
and apache-airflow-providers-standard==0.0.1.dev0 depends on
apache-airflow-providers-common-sql>=1.20.0.dev0, we can conclude that
apache-airflow-providers-standard==0.0.1.dev0 cannot be used.
And because only apache-airflow-providers-standard==0.0.1.dev0 is
available and you require apache-airflow-providers-standard, we can
conclude that your requirements are unsatisfiable.
Traceback (most recent call last):
File "/opt/airflow/scripts/in_container/install_airflow_and_providers.py", line 565, in
install_airflow_and_providers()
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/rich_click/rich_command.py", line 152, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "/opt/airflow/scripts/in_container/install_airflow_and_providers.py", line 541, in install_airflow_and_providers
run_command(base_install_providers_cmd, github_actions=github_actions, check=True)
File "/opt/airflow/scripts/in_container/in_container_utils.py", line 47, in run_command
result = subprocess.run(cmd, **kwargs)
File "/usr/local/lib/python3.9/subprocess.py", line 528, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/local/bin/uv', 'pip', 'install', '--python', '/usr/local/bin/python', '/dist/apache_airflow-3.0.0.dev0.tar.gz', '/dist/apache_airflow_providers_standard-0.0.1.dev0.tar.gz', '/dist/apache_airflow-3.0.0.dev0.tar.gz']' returned non-zero exit status 1.
```
We should be a bit smarter here and detect (in each chunk separately) which providers are **additionally** needed to be built - in this case since standard depends on `comon.sql >= 1.20.0`, the sdist chunk installation should add common.sql to the list of installed providers in the chunk where `standard` is installed.
Contributor guide
Research direction
Start with scripts/in_container/install_airflow_and_providers.py, especially the provider installation path shown in the traceback, and trace how sdist build chunks select providers. Reproduce the dependency failure from PR 44018 and inspect the related parallel build flow. Done means each chunk includes any additionally required provider versions so dependency resolution succeeds.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- build-system, ci-cd
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100