Safe multi-proxy TUS chunk upload (BA-3974 implementation)
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 15h 13m
- Merged PRs (30d)
- 368
Description
## Background
Reported in **BA-3974**: when multiple Storage Proxy replicas share an NFS mount, concurrent PATCH requests for the same TUS upload session corrupt data because every replica appends to one temp file under the same path. The BA-3678 fix (offset header check) only protects against single-client offset errors and does not address the cross-replica race.
## Goal
Replace the single-file append model with a metadata-driven per-offset chunk store, fcntl-locked metadata, sha256 checksums, atomic rename, and a recovery-friendly status endpoint. Wire format on /upload (PATCH/HEAD/OPTIONS) stays TUS 1.0.0-compatible so existing tus-js-client based callers work unchanged.
## Scope (6 stacked PRs)
1. Upload session state model (data classes + errors + pure unit tests)
1. TusUploadSession storage class (fcntl lock, atomic rename, sha256, concurrency tests)
1. Rewire TUS PATCH/HEAD to chunk-based store (the user-visible bug fix)
1. Multi-proxy NFS race regression test (reproduces the bug via stat() mock)
1. TUS Checksum extension support (Upload-Checksum header, HTTP 460)
1. /upload/status endpoint + progress headers (resumable recovery)
## Acceptance
- All 6 sub-tasks merged.
- Unit tests cover the chunk store and the HTTP handlers under concurrent workloads.
- Existing tus-js-client based clients (Web UI) continue to work without code changes.
- BA-3974 is verified fixed (multi-proxy concurrent chunks no longer corrupt data).
JIRA Issue: BA-6153
Contributor guide
Research direction
Start by tracing the /upload PATCH, HEAD, and OPTIONS handlers, then review the stated session-state, chunk-store, concurrency, checksum, and /upload/status requirements. Use the planned pure unit tests, concurrency tests, HTTP handler tests, and stat() mock regression test to verify completion; done means all six sub-tasks are merged and existing tus-js-client callers remain compatible.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend-api-design, distributed-systems, testing
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100