daphne-project / daphne-project/daphne
Containers with TensorFlow and PyTorch
- Dominant language
- C++
- Stars
- 81
- Forks
- 83
- PR merge metrics
- No merged PRs in 30d
Description
Since today, DaphneLib supports efficient data exchange with TensorFlow and PyTorch. Currently, they are optional dependencies, i.e., if they are installed, the related DaphneLib features can be used, if they are not installed, DaphneLib still works, but the TF/PT-related features don't. Moreover, the test cases for the data exchange with TF/PT are only run when these libraries are installed.
Hence, users and developers are not forced to install TF/PT unless they really want to interoperate with them or run the respective test cases. That also means, **there is no urgent problem**.
Currently, the DAPHNE container images do not contain TF/PT. To provide users the opportunity to play around with the data exchange with TF/PT and to allow developers (or the CI) to run *all* test cases (including the ones that require TF/PT), we could add TF/PT to our containers by:
```bash
pip install tensorflow torch
pip install --upgrade pandas
```
Note that the version of pandas currently available in the containers (installed via `apt`) is too old.
However, TF/PT are quite large libraries. I built a container including them on my system and it has a size of ~14 GB, while the current `daphne-dev` container image is just ~4 GB. We should not put the burden of downloading such a large container image on every user/developer. Thus, we could discuss the introduction of a separate container image that contains TF/PT (maybe similar to how it's already done for CUDA). For the CI, it would be great to run an image that has TF/PT, such that all test cases execute.
Contributor guide
Research direction
Start by reviewing the repository's current container-image definitions and CI configuration, then determine how the existing CUDA image arrangement works. Clarify whether the project wants a separate TensorFlow/PyTorch image and which CI tests should use it; the work is done when that image and its CI usage are agreed and implemented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, pandas, pytorch, tensorflow
- Domain
- ci-cd, devops, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100