lablup / lablup/backend.ai

Add TusUploadSession storage class

Open
#11,760 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

## Scope

Implement the async storage class that owns the on-disk session layout (info.json + chunks/chunk_.dat + .lock). All concurrency-safety properties live here.

## Deliverables

- `src/ai/backend/storage/services/upload/session.py` — TusUploadSession class: ensure_initialized, read_state, open_temp_chunk, commit_chunk (with fcntl.flock + atomic rename), assemble, cleanup.
- stream_chunk_to_temp helper for streaming a request body into a temp chunk file with sha256 computation.
- Tests: lifecycle, sequential, out-of-order, idempotent replay, conflict, assembly, cleanup, JSON corruption, ThreadPoolExecutor-based concurrency (disjoint offsets and same-offset).

## Acceptance

- Exactly one committer wins per offset under concurrent same-offset attempts; others observe idempotent replay or ChunkConflictError.
- info.json updates are atomic (tmp + rename).
- completed_now flag fires exactly once per session.
- pants fmt/lint/check/test passes.

## Depends on

BA-3974 [1/6] (state model).

JIRA Issue: BA-6155

Contributor guide

Open the contributing guide

Research direction

Start with the BA-3974 state model, then inspect src/ai/backend/storage/services/upload/session.py and the surrounding upload storage entry points. Use the lifecycle, ordering, replay, conflict, assembly, cleanup, corruption, and ThreadPoolExecutor concurrency tests as the guide. Done means the listed Pants checks pass and the acceptance guarantees hold.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.