lablup / lablup/backend.ai

Safe multi-proxy TUS chunk upload (BA-3974 implementation)

Open
#11,758 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

## Background

Reported in **BA-3974**: when multiple Storage Proxy replicas share an NFS mount, concurrent PATCH requests for the same TUS upload session corrupt data because every replica appends to one temp file under the same path. The BA-3678 fix (offset header check) only protects against single-client offset errors and does not address the cross-replica race.

## Goal

Replace the single-file append model with a metadata-driven per-offset chunk store, fcntl-locked metadata, sha256 checksums, atomic rename, and a recovery-friendly status endpoint. Wire format on /upload (PATCH/HEAD/OPTIONS) stays TUS 1.0.0-compatible so existing tus-js-client based callers work unchanged.

## Scope (6 stacked PRs)

1. Upload session state model (data classes + errors + pure unit tests)
1. TusUploadSession storage class (fcntl lock, atomic rename, sha256, concurrency tests)
1. Rewire TUS PATCH/HEAD to chunk-based store (the user-visible bug fix)
1. Multi-proxy NFS race regression test (reproduces the bug via stat() mock)
1. TUS Checksum extension support (Upload-Checksum header, HTTP 460)
1. /upload/status endpoint + progress headers (resumable recovery)

## Acceptance

- All 6 sub-tasks merged.
- Unit tests cover the chunk store and the HTTP handlers under concurrent workloads.
- Existing tus-js-client based clients (Web UI) continue to work without code changes.
- BA-3974 is verified fixed (multi-proxy concurrent chunks no longer corrupt data).

JIRA Issue: BA-6153

Contributor guide

Open the contributing guide

Research direction

Start by tracing the /upload PATCH, HEAD, and OPTIONS handlers, then review the stated session-state, chunk-store, concurrency, checksum, and /upload/status requirements. Use the planned pure unit tests, concurrency tests, HTTP handler tests, and stat() mock regression test to verify completion; done means all six sub-tasks are merged and existing tus-js-client callers remain compatible.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend-api-design, distributed-systems, testing
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.