lablup / lablup/backend.ai

Create bai CLI E2E scenario test scripts for core user journeys

Open
#11,283 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

Create a scenario test package at the repo root with shell scripts that exercise core user journeys via the bai CLI.

Domains: session, model service (deployment), model card, vfolder
Format: shell scripts combining bai CLI commands into end-to-end user journey flows
Location: scenarios/ at repo root

## Scenarios

### Setup (common)
Admin creates users, keypairs, resource policies, projects. All subsequent scenarios run as the created user(s).

### 1. VFolder Lifecycle
create → my-search (confirm) → mkdir → upload → ls → download → mv → rm → delete → my-search (confirm deleted)

### 2. Session Lifecycle
vfolder create → session enqueue (vfolder mount) → project-search (wait RUNNING) → get → logs → start-service → shutdown-service → terminate → project-search (confirm terminated)

### 3. Model Card → Deploy
vfolder create (model) → upload → model-card project-search (confirm) → available-presets → deploy → deployment project-search (confirm) → replica search (healthy) → access-token create → inference request → deployment delete → deployment project-search (confirm deleted) → vfolder delete

### 4. Deployment + Revision Management
vfolder create (model) → upload → deployment create → deployment project-search → revision add → revision search → revision activate → revision current → replica search → deployment update (replicas) → replica search (scale confirm) → deployment delete → vfolder delete

### 5. Teardown Verification
Create multiple resources across domains → search each domain (confirm exist) → delete in reverse order → search each domain (confirm deleted)

### 6. Multi-user Access Control
admin creates userA, userB (same project) → userA: vfolder create, session enqueue, deployment create → switch to userB: search userA resources (project-shared visible?) → attempt access to userA private resources (expect denied) → cleanup

### 7. VFolder Invite + Clone
userA: vfolder create → invite userB → userB: ls/download invited vfolder (access confirmed) → upload attempt (permission level check) → userB: vfolder clone → my-search (confirm owned) → userA: delete original → userB: clone still accessible → cleanup

### 8. Cross-project Isolation
admin creates projectA, projectB → userA(projectA), userB(projectB) → userA: vfolder create, deployment create → userB: search projectA resources (must not see) → userB: create own resources in projectB → each user sees only own project → cleanup

## Scenario Coverage Matrix

|1. | Scenario | VFolder | Session | Model Card | Deployment | Access Control |
|---|---|---|---|---|---|---|
|—|----------|---------|---------|------------|------------|----------------|
| 1 | VFolder Lifecycle | CRUD+files |
- |
- |
- |
- |
| 2 | Session Lifecycle | mount | CRUD+service |
- |
- |
- |
| 3 | Model Card → Deploy | model upload |
- | search+deploy | CRUD+token |
- |
| 4 | Deployment + Revision | model upload |
- |
- | revision+scale |
- |
| 5 | Teardown Verification | delete | terminate |
- | delete |
- |
| 6 | Multi-user Access | O | O |
- | O | same project |
| 7 | VFolder Invite+Clone | invite+clone |
- |
- |
- | user sharing |
| 8 | Cross-project Isolation | O |
- |
- | O | project isolation |

## Success Criteria
- [ ] scenarios/ directory created at repo root with shell scripts
- [ ] Admin user creation as setup step, all tests run as regular user
- [ ] Scenarios 1-4: single-user domain lifecycle with search verification at each step
- [ ] Scenarios 6-8: multi-user and cross-project access control verification
- [ ] README.md with scenario coverage matrix
- [ ] Scripts are runnable against a live Backend.AI cluster

JIRA Issue: BA-5834

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.