lablup / lablup/backend.ai

Add multi-proxy NFS race regression test

Open
#11,762 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

## Scope

Lock in the BA-3974 regression with a unit test that reproduces the cross-replica corruption on a local filesystem by mocking Path.stat (to simulate NFS attribute cache staleness) and using lseek+write (to simulate the lack of cross-client O_APPEND atomicity). The same workload is then driven through the new TusUploadSession to prove byte-perfect output.

## Deliverables

- `tests/unit/storage/services/upload/test_nfs_race_reproduction.py`
- Two test classes: TestLegacyModelCorruptsUnderStaleStatCache (proves the bug existed) and TestNewModelSurvivesSameChaos (proves the fix works under identical workload).

## Acceptance

- Legacy reconstruction reliably corrupts the file under N concurrent workers with frozen stat.
- New model assembles a byte-perfect output even with 3x duplicate retries per chunk fired concurrently.

## Depends on

BA-3974 [3/6] (handler rewire).

JIRA Issue: BA-6157

Contributor guide

Open the contributing guide

Research direction

Start with tests/unit/storage/services/upload/test_nfs_race_reproduction.py and trace the existing upload model and new TusUploadSession used by the test. Run the regression test to verify that frozen Path.stat with concurrent lseek+write workers corrupts the legacy model while the new model produces byte-perfect output with three duplicate retries per chunk.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, distributed-systems, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.