lablup / lablup/backend.ai

Perform small Test Workload for image verification

Open
#3,136 0 comments 0 reactions 1 assignee Claimed by @jopemachine View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

### Main idea

- Run a small test workload using the image (e.g., fashion-MNIST for CUDA-enabled images)\* We start simple, but it's important to set up an automated workflow so that we could add more sophisticated test workloads as we find new compatibility issues in our customer sites and as new CUDA versions & AI frameworks come out.

#### Implementation ideas dump

I have focused on designing a scalable and easily extensible approach for automated image testing.

For example, tests should be easy to add, remove, or update, and each image should be able to use different test workloads or, conversely, share the same test workloads.

- Create an image label named `ai.backend.test-resource-location` and store the URL or the user's local path of the test workload file in the label.

- Create a command like `backend.ai mgr image run-test ` performing the following tasks sequentially.

1. Download the test workload from the `ai.backend.test-resource-location`.
1. Create a vfolder containing the test workload.
1. Create a session using the given image.
1. Mount the vfolder to pre-specified path into the created session once the session is ready.

- Create a kernel runner operation like `run-image-test` and its handler executes the test workload from the mounted VFolder. (For example, the simplest approach would be to consider executing a shell script using `asyncio.create_subprocess_exec`.)

### Alternative ideas

_No response_

### Anything else?

_No response_

JIRA Issue: BA-50

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.