Free-threading support

This is a meta-issue for what Dask needs in order to declare compatibility with Python 3.14t:

• The handling of warnings is substantially different in free-threading interpreters.
  • dask: (https://github.com/dask/dask/pull/12224)
  • distributed: WIP
• contextvars are propagated automatically when spawning threads. distributed could use a PR to simplify its handling.
• dask/dask CI: https://github.com/dask/dask/pull/12284
• dask/distributed CI: https://github.com/dask/distributed/pull/9194 tests are failing and need investigating. I do not know at this point if they highlight any actual issue.
• dask/distributed CI: we could use some more aggressive multithreading tests for a few common uses cases:
  • different threads run in parallel in the client process, each with their own Client, sharing the same Scheduler
  • different threads run in parallel in the client process, sharing a single Client
  • different threads wait on the same Future (this is a variant of the previous point)
• pandas: Need Pandas >=3.0.1
• msgpack: needs >=1.2.0rc1 to release the GIL
• brotli: waiting for release >=1.2.1 (https://github.com/google/brotli/issues/1241). Optional dependency of urllib3. It is pulled in by the distributed conda-forge recipe. Workarounds:
  1. use pip to install dask and distributed
  2. force-uninstall the brotli conda package
• cytoolz: The package was not properly audited AFAIK, but it should be ok thanks to its stateless nature, as long as you don't do obviously dangerous stuff like having multiple threads draw from the same generator.
• investigate if it's possible to tweak pip and conda-forge dependencies for free-threading only, and if so set much more stringent minimum versions.
• performance regression testing at scale vs. 3.14 with GIL
• documentation and user announcement

The above are the prerequisites to advertise a good user experience to final users - e.g. it will be at worst like the GIL-enabled version, minus some % slower due to known bottlenecks that are actively being ironed out upstream.

In addition, real world users will be blocked by key missing optional packages; notably:

• zarr (blocked by its dependency numcodecs: https://github.com/zarr-developers/numcodecs/issues/749)
• h5py (for xarray's NetCDF engine="h5netcdf")
• s3fs (in conda, it pulls in brotli through urllib3; see above for workarounds)
• tiledb
• jupyterlab (you can run jupyterlab in GIL-enabled Python and then attach a 3.14t kernel, but it's fiddly)

After the above, there is ample margin for performance tweaking:

• pickling/unpickling runs on a special thread pool with a single thread. With free-threading, you could have as many threads as you have CPUs.
• network I/O runs on a single thread; it could be moved to one thread per peer.
• spilling is single-threaded, blocks the worker state machine and networking, and has always been very painful
• the only thing that must remain single-threaded is the state machine, but I expect it not to be a bottleneck in and by itself.
• more in general, there is a lot of profiling needed as it becomes possible to have workers with a huge number of threads - which was previously unadvisable even at the default chunk size (128 MiB). My latest benchmarks at scale (on Coiled, circa 2022) showed severe performance degradation already when moving from 4 threads per worker to 8, given the same number of total CPUs on the cluster.

This is a meta-issue rather than a single code task. Start with the failing distributed CI work linked in pull request 9194 and the unchecked warnings-handling item, then review each remaining prerequisite and optional-package blocker. Done means the compatibility checklist is resolved well enough to announce Python 3.14 free-threading support, with performance and documentation work addressed.