Perform small Test Workload for image verification
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 17h 7m
- Merged PRs (30d)
- 358
Description
### Main idea
- Run a small test workload using the image (e.g., fashion-MNIST for CUDA-enabled images)\* We start simple, but it's important to set up an automated workflow so that we could add more sophisticated test workloads as we find new compatibility issues in our customer sites and as new CUDA versions & AI frameworks come out.
#### Implementation ideas dump
I have focused on designing a scalable and easily extensible approach for automated image testing.
For example, tests should be easy to add, remove, or update, and each image should be able to use different test workloads or, conversely, share the same test workloads.
- Create an image label named `ai.backend.test-resource-location` and store the URL or the user's local path of the test workload file in the label.
- Create a command like `backend.ai mgr image run-test ` performing the following tasks sequentially.
1. Download the test workload from the `ai.backend.test-resource-location`.
1. Create a vfolder containing the test workload.
1. Create a session using the given image.
1. Mount the vfolder to pre-specified path into the created session once the session is ready.
- Create a kernel runner operation like `run-image-test` and its handler executes the test workload from the mounted VFolder. (For example, the simplest approach would be to consider executing a shell script using `asyncio.create_subprocess_exec`.)
### Alternative ideas
_No response_
### Anything else?
_No response_
JIRA Issue: BA-50
Contributor guide
Assessment
This issue has not been assessed yet.