apache / apache/airflow

Sphinx cannot be upgraded to latest version because it has Python 3.11 as minimum

Open
#52,513 0 comments 0 reactions 0 assignees View on GitHub
area:dependencies
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

We likely shoudl not even attempt to solve it - before 3.11 is the "lowest" version. Dumping the result of the discussion we held in Slack on why 3.10 + CI Image + Sphinx are mostly coupled and what effects it might have if we want to run docs building on 3.11.

For example running mypy on 3.10 and 3.11 will 100% produce different results
And we are using the very same "default" 3.10 image to run our mypy checks
So this is a "reproducibility" feature and "optimization" why all our code and images are by default "floored" to the lowest possible version

We might think of doing it differently - but we will have to compromise on "reproducibility" and "optimization"
and those two are super important (and my personal priority) taking into account 3600+ contributors we have - making their life harder because we want to upgrade sphinx, is "no-go" for me

I mean - surely - if someone comes with a solution that does not compromise those two - super
but I was unable to come with one (and I tried) - "reproducibilty" and "optimisation" is top 1

If we find a way how to upgrade sphinx without compromizing those - cool

And staying on "default" (i.e. lowest) version of Python for everything that involves Python code and requires CI image, is the best way I found to keep those two.
But in the last 4 years I was unable to find such a solution

So it's likey it's not worth even spending time on discussing it

One of the big problems is space used on local machines. Currently in order to build docs Sphinx is importing a lot of code from providers, and it means that it really has to be run on the system that has all dependencies installed - i.e. inside the CI image. Running it on different architectures and OS-es might simply fail. This is the main reason why "breeze build-docs" uses the image - because of "reproduicibility". And yes -we could build separately "3.11" image for it for every developer - but i am not sure if you noticed but with uv and cache, breeze recently started to use a lot, a lot of space - because uv aggressively caches things and we have many "python-version" dependent libraries that are build and cached by UV in different versions - one locally and one per image (because in the image there are different system libraries the python compiled code is linked to) - so this means that if you want to keep the 3.10 image (if only for mypy) - you also need to keep 3.11 image (on it's own, includin build artifacts it's likely > 10GB ) + all UV cache (2x) for 3.11 python version librarires (Also multiple GB)

So for "optimisation's" sake - we really want majority of People and our CI to use mostly one image -> and since using 3.11+ image might involve introducing 3.10 incompatibilities by contributors -> 3.10 is an obvious choice

I recently started to run out of disk on my 1TB linux - mostly because I do often switch between versions and images - and I have to do cleanup more aggressively.

We certainly have a ton of contributors who have even less disk available

We should take all those consequence into consideration -> because in this case "upgrading to latest sphinx" means all that will be impacted. This is what I mean by weighing "pros" and "cons"

Contributor guide

Open the contributing guide

Research direction

Start with the issue's discussion of `breeze build-docs`, the default Python 3.10 CI image, and Sphinx's dependency imports. A useful resolution would need to show whether Sphinx can be upgraded without sacrificing reproducibility, disk usage, or the Python-version baseline; the issue provides no files or tests to run.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system, ci-cd, documentation
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.