Switch packaging to PDM/Hatch/Poetry

There's some discussion in #2681, and some more in Gitter, but let's make a dedicated issue for this.

Alternatives:

1. Status quo
   • dependabot is a hassle
   • sort of impossible to properly define different package version requirements on different versions
   • regenerating *-requirements.txt locally, as is currently required to pass CI whenever changing something in *-requirements.in, requires having a matching python version and possibly other requirements on your environment.
2. Poetry
   • Does not comply with PEP 621: https://github.com/python-poetry/roadmap/issues/3
   • there's an open PR for switching to Poetry: #2595
   • Supports lockfiles, quoting @njsmith on gitter:

     lockfiles for a library are very valuable for keeping CI functioning
     you don't ship them, but if you don't have them in CI and don't have very rapid development, then every time you do try to merge a PR you discover that actually CI has been broken for a while and no-one noticed so now you have to untangle that before you can finish your original thought

3. Hatch

   • Does not support lockfiles, but there are different ways to get around the above problem
     quoting @apollo13

     Fair enough. But with a nightly run you would also notice if a dependency update is breaking stuff and you could react faster by limiting the dependency till you fixed it

     quoting @agronholm

     what I do in my projects instead is cache the dependencies
     so they only get updated if I update my pyproject.toml or manually delete the caches
     => no constant dependabot churn

4. PDM
   • supports lockfiles

Start by reading the discussion in #2681, the Gitter discussion, and the existing Poetry pull request #2595. Compare the listed status quo, Poetry, Hatch, and PDM alternatives; work is done only once the project agrees on a packaging approach and the migration scope is defined.