[TECH DEBT] proper dependency listing (migrate to poetry or equivalent)

Description of the tech debt to be addressed, include links and screenshots

I'm trying to to fix saltstack package in nixpkgs (https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/admin/salt/default.nix)

And I'm running into all kinds of issues, things like:

• some optional dependencies are mandatory on some platform like darwin (https://github.com/saltstack/salt/blob/v3004.1/requirements/darwin.txt) while they are not mandatory for linux:
  • libcloud
  • cherrypy
  • gitpython
  • etc
• some dependencies are conflicting, because of code that is parsing them doesn't work correctly, for example got pyzmq at the same time:
  • pyzmq<=20.0.0
  • pyzmq>=17.0.0
  • pyzmq>19.0.2
• there are some other issues as well, although as I'm writing this I'm not 100% sure they are salt issue, so I'll skip them to not cause confusion.

I understand that you want to match salt to what's available for each platform but it appears to be done incorrectly. Perhaps due to limitation of setuptools when work on saltstack begun. At this point though I believe that dependency problem has been largely solved.

I mostly use poetry so my recommendations would revolve around it, but I think you should be able to even use setuptools (but if you do please use declarative config defined in setup.cfg)

You can still specify base dependencies with ranges of versions that are supported. Add environment markers (which you use in requirements.txt files already to make dependency required only for specific platforms). List optional dependencies as extras and also specify dev dependencies (to run tests or formatting, which also could be customized via extras).

https://python-poetry.org/docs/pyproject/#dependencies-and-dev-dependencies
https://python-poetry.org/docs/pyproject/#scripts
https://python-poetry.org/docs/pyproject/#extras
https://python-poetry.org/docs/dependency-specification/

I understand that there might be need o provide locks for different platforms and python version, but you could provide multiple poetry.lock or generate requirements.txt for each although ideally this should be avoided and version ranges should just cover all versions that supposed to work.

I created this ticket as a tech debt, because I believe the current state contributes to decreased reliability of saltstack as it looks like a nightmare to update all files and is prone to errors.

Versions Report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)
Not relevant.

Start by comparing requirements/darwin.txt with the other requirements files and review the proposed Poetry or declarative setuptools configuration. Define how platform-specific, optional, and development dependencies should be represented, then verify that supported installations no longer require conflicting or incorrectly mandatory dependencies.