lablup / lablup/backend.ai

Integrated tests for kernels

Open
#81 0 comments 0 reactions 1 assignee Assigned to @inureyes View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

This issue replaces lablup/backend.ai-kernels#3 and represents the work in [lablup/backend.ai-kernel-tester](https://github.com/lablup/backend.ai-kernel-tester).

The goals:

- Execute realistic/real code samples (with minimized input data) by collecting actual error cases.
- Run them in a fully integrated setup: manager + agent + accelerator plugin.\* For now, we assume that the user of the test suite already has a working setup.

- Make it possible to reuse a code sample across multiple different kernels if the kernels contain the same set of dependency packages for the sample.
Future goals:

- Automate execution of this integration test suites upon every push to the `backend.ai-kernels` repository, only selectively by running tests related to the commit changes.
- Prepare a high-performance kernel image builder farm (Azure Pipelines? on-prem?)
- Make it possible to run the same tests using different version combinations of the manager/agent/accelerator plugins/Docker/OS, in an automated way. (Vagrant + Ansible?)\* It is not practical to run all possible combinations as CI for every push. For CI, just keep a few representative (stable-based) combinations, but let's allow customization of the combinations with automated setup.

┆Issue is synchronized with this [Asana task](https://app.asana.com/0/1159751085623729/1159756855629509) by [Unito](https://www.unito.io/learn-more)

```Java
### Tasks
- [ ] https://github.com/lablup/backend.ai/issues/1089
- [ ] https://github.com/lablup/backend.ai/issues/1088
```

JIRA Issue: BA-316

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.