lablup / lablup/backend.ai

Restore the tester integration suite to run against current server contracts

Open
#13,236 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

## Objective

Bring the tester suite in `src/ai/backend/test` back to a green baseline. It defines 82 specs (91 cases after parametrization; 42 specs / 49 cases in the default `backend.ai test run`), but most of them no longer pass against the current manager. Deliverable: a default local dev environment where `backend.ai test run` is green, and every spec that needs setup beyond that environment is tagged and excluded by the default run config.

## Background

The framework itself is healthy: all modules import, and every client SDK method and GraphQL field the suite uses still exists. Verified against a live local dev environment - `login`, `creation_interactive_session_failure_low_resources`, `creation_interactive_session_failure_too_many_container_count` and `purge_group_success` pass. Everything else fails on two server contract changes, one unset configuration value, and one cleanup bug.

## Scope

### Blockers

- Session event SSE no longer tolerates listen-before-create (17 specs). `GET /events/session` resolves the session name to an ID up front and raises `SessionNotFound` when the session does not exist yet, while the session templates start the listener before calling `get_or_create`. Candidate fix: create with `enqueue_only`, then subscribe by session ID, which also makes the PENDING to RUNNING transition an explicit assertion.
- VFolder creation requires a group UUID (10 specs). `VFolderCreateReq.group_id` is a UUID, but the tester config passes the group name. The client SDK `VFolderByName.create` still declares group as a plain string and forwards it verbatim, so this path is broken outside the tester too.
- Model service specs cannot run unconfigured (6 specs). `configs/tester/tester.toml` carries a placeholder `model-vfolder-name`, yet the single-node single-container model service specs are not excluded from the default run. `ModelVFolderTemplate` exists but is wired to no spec.
- Sessions leak when creation verification fails. `InteractiveSessionTemplate` skips the destroy step when `_verify_session_creation` raises after the session was actually created, so every failing run leaves a container behind.

### Secondary cleanups

- `InteractiveSessionTemplate` omits the architecture argument while the batch and failure specs pass it from config, so the two request different images from the same configuration.
- Spec key and name disagree for the commit spec, so `run-test` cannot find it by its declared name.
- The session status history expected set omits `RESERVED`, `DEPRIORITIZING`, `RESCHEDULING`, `PREEMPTED`, `RESTARTING` and `RUNNING_DEGRADED`.
- Dead code wired to no spec: `ModelVFolderTemplate`, `VFolderPurgeSuccess`, `VFolderDeleteAndRestoreSuccess`, `SequenceTestTemplate`. VFolder purge and restore are worth keeping as specs.
- `src/ai/backend/test/README.md` points at `sample-env-tester.sh`, which no longer exists in the repository.

## Acceptance Criteria

- `backend.ai test run` passes on a default local dev environment, or every remaining failure belongs to a spec explicitly tagged as requiring extra setup.
- No compute session, vfolder, user or group is left behind after a run, including runs that fail mid-way.
- Specs that need setup beyond the default dev stack carry a required-configuration tag and are excluded by the default run config.
- Orphaned templates and testcases are either wired into specs or removed.
- Each child issue is independently completable and verifiable, and estimated at 2 points or less.

## References

- `src/ai/backend/test/README.md` and `src/ai/backend/test/tester/README.md`
- Related epic: BA-7033 Full-stack end-to-end integration test framework across all components

JIRA Issue: BA-7069

Contributor guide

Open the contributing guide

Research direction

Start with src/ai/backend/test/README.md, src/ai/backend/test/tester/README.md, and the tester entry point used by `backend.ai test run`; map the listed templates and specs to their setup requirements. Run the default suite in a local development environment, then verify that contract failures, cleanup failures, orphaned templates, and configuration-dependent specs are addressed. Done means the default run is green or explicitly excludes tagged extra-setup specs, with no resources left behind.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend-api-design, testing
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.