[Feature]: Restructure python dependency deployment model

Description

The DUNE DAQ org is currently entirely dependent on fixed versions of python packages, and in preparation for containerized deployments of certain applications (such as the process_manager, which is ready for such deployment), a more flexible dependency versioning model is required. Specifically, the following limitations/requests are encountered:

• When deploying an application, even when using only a subset of the full fixed monolithic virtual environment, it must be used in its entirety. For deploying individual applications, only a subset of the monolithic venv are required, allowing for a more lightweight deployment.
• All package versions are fixed, even if the relevant packages are not used, restricting the use of more modern dependencies which would otherwise be possible for monolithic venv functionality. As containerized application deployments will not require the full venv, they should have more flexibility in dependency versioning, which should be propagated throughout the python packages more globally.

It is thus required that we update the dependency deployment model to align with longer term strategies.

Potential impact radius

Large

Reason for change

Defined above.

Suggested implementations

The venv we use is large and complex, using a combination of in-venv dependencies and some spack deployed dependencies. A significant portion of this will be refactoring many of these repositories to be simpler. As an example, the kafkaopmon and erskafka repos are both mixed case repos (C++ and python), which may require splitting out in order to achieve this flexibility.

Testing suggestions

Once we have this deployment model clarified, we should run all standard tests, both in software only but also in hardware.

Anything else?

No response

Start by mapping the monolithic virtual environment, its in-venv and Spack dependencies, and the deployment needs of process_manager. Review kafkaopmon and erskafka as examples of mixed C++ and Python repositories, then clarify the deployment model before running the standard software-only and hardware tests.