lablup / lablup/backend.ai

Introduce deep-learning test automation procedure to create a stable environment

Open
#112 0 comments 0 reactions 1 assignee Assigned to @achimnol View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

**Is your feature request related to a problem? Please describe.**

As the number of kernels managed by Backend.AI increases, the stability of each kernel becomes a major issue. In some cases, the stability of the kernel environment itself is inherent, but the libraries, code, and application software in the kernel often collide with each other. Currently, after a simple operation test, we respond whenever an issue occurs, but the frequency of problems is increasing in a rapidly changing deep learning framework environment. In addition, since the introduction of custom kernel support, user-defined kernels have been increasing, and if a problem occurs in these kernels, it is difficult to find the cause of the problem. In this case, the standardized test method aims to improve the overall user experience by introducing a method of pre-testing the stability of each kernel at the time each kernel is built and deploying the stabilized kernels.

In Backend.AI, since most kernel environments are software targeted at high-performance computing environments optimized for deep learning or research environments, it is difficult to detect problems with general project or software unit tests. Therefore, not only various kinds of software environments but also CPU, RAM, and various GPU environments must be tested for kernel testing. In addition, even in the case of GPUs, testing for the minimum necessary GPU resources must be performed before a proposal to start the corresponding kernel environment can be made.

**Describe the solution you'd like**

Therefore, a stable software working environment is introduced through the following improvements.

- [x] 1. Software that can automatically test with Backend.AI while making scalable changes to CPU, RAM and GPU
- [ ] 2. Test hardware and Backend.AI environment where the software can run @xyloon
- [ ] 3. A basic set of tests that can be applied commonly depending on the kernel @hephaex
- [ ] 4. Introduce specific tests to run tests on specific kernels @gofeel @inureyes
- [ ] 5. Apply continuous updates to test hardware / environment @gofeel
- [ ] 6. Software routines or shell scripts, including the kernel build process, that force builders to run tests after building as you build a new kernel @inureyes
- [x] 7. Docker container registry for storing and managing test images @achimnol
- [ ] 8. Custom tag / domain rule to separate test images from normal kernel images in custom docker registry @achimnol
In addition, the difference in kernel management enables the separation of a stable kernel-based production environment from an unstable modern kernel environment that continues to introduce new software and environments. The necessary contents for this are as follows.

- [x] 1. Establish a kernel management system rule. The kernel is differentiated between the fully managed kernel, the community kernel, and the deprecated kernel. @inureyes
- [ ] 2. Post a kernel management system rule decided at 1 @inureyes
- [ ] 3. Create a list of kernels according to the scheme. The list of kernels managed at this point is sorted and classified according to the above scheme. @hephaex
- [ ] 3. Create kernel Test Sets and Environments for Managed Kernels @hephaex @gofeel @inureyes
For each of the above items, create an issue in the corresponding project and repository and proceed.

**Describe alternatives you've considered**

**Additional context**

┆Issue is synchronized with this [Asana task](https://app.asana.com/0/1159751085623729/1159756812379669) by [Unito](https://www.unito.io/learn-more)

JIRA Issue: BA-310

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.