daphne-project / daphne-project/daphne

Containers with TensorFlow and PyTorch

Open
#705 2 comments 0 reactions 0 assignees View on GitHub
CI/CD DaphneLib Discussion
Dominant language
C++
Stars
81
Forks
83
PR merge metrics
No merged PRs in 30d

Description

Since today, DaphneLib supports efficient data exchange with TensorFlow and PyTorch. Currently, they are optional dependencies, i.e., if they are installed, the related DaphneLib features can be used, if they are not installed, DaphneLib still works, but the TF/PT-related features don't. Moreover, the test cases for the data exchange with TF/PT are only run when these libraries are installed.

Hence, users and developers are not forced to install TF/PT unless they really want to interoperate with them or run the respective test cases. That also means, **there is no urgent problem**.

Currently, the DAPHNE container images do not contain TF/PT. To provide users the opportunity to play around with the data exchange with TF/PT and to allow developers (or the CI) to run *all* test cases (including the ones that require TF/PT), we could add TF/PT to our containers by:
```bash
pip install tensorflow torch
pip install --upgrade pandas
```
Note that the version of pandas currently available in the containers (installed via `apt`) is too old.

However, TF/PT are quite large libraries. I built a container including them on my system and it has a size of ~14 GB, while the current `daphne-dev` container image is just ~4 GB. We should not put the burden of downloading such a large container image on every user/developer. Thus, we could discuss the introduction of a separate container image that contains TF/PT (maybe similar to how it's already done for CUDA). For the CI, it would be great to run an image that has TF/PT, such that all test cases execute.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the repository's current container-image definitions and CI configuration, then determine how the existing CUDA image arrangement works. Clarify whether the project wants a separate TensorFlow/PyTorch image and which CI tests should use it; the work is done when that image and its CI usage are agreed and implemented.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, pandas, pytorch, tensorflow
Domain
ci-cd, devops, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.