TraceMachina / TraceMachina/nativelink
CAS uploads succeed but stored files are incomplete due to async rename race condition
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 1.6k
- Forks
- 244
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 54
Description
When using the LUCI Go CAS client (cas.exe) to archive .isolate files, all resources appear to be uploaded successfully to the CAS server. However, the uploaded files on the server side are incomplete (missing data).
After investigating, the issue seems to originate from the NativeLink CAS server implementation, specifically in:
nativelink-store/src/filesystem_store.rs
Observed flow:
update_oneshot()
→ write temp file ✅
→ sync ✅
→ drop file ✅
→ call emplace_file()
→ spawn background task (rename not finished yet ❗)
→ return OK 🚨
Meanwhile, on the client side:
cas.exe
→ starts downloading immediately
→ calls get_part_unchunked()
Problem:
There is a race condition between upload completion and file availability:
update_oneshot() returns OK before the file is fully moved (renamed) to its final location.
The emplace_file() function spawns a background task to perform the rename, but does not wait for it to complete.
The client begins downloading immediately after receiving success.
At this point, the file may still be in the temporary path or the rename operation is not finished.
Result:
Clients may read incomplete or partially written files from the CAS server.
Expected behavior:
The server should only return success after:
The file has been fully written,
Synced to disk,
And atomically moved (rename completed) to its final location.
Suggested fix:
Ensure the rename operation in emplace_file() is completed synchronously before returning success,
or
Introduce a mechanism to guarantee file visibility/consistency before allowing reads.
Environment:
LUCI Go CAS client (cas.exe)
NativeLink CAS server
Let me know if more logs or a minimal reproduction are needed.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read nativelink-store/src/filesystem_store.rs, starting with update_oneshot() and emplace_file(), then reproduce the upload and immediate download flow with the LUCI Go cas.exe client and get_part_unchunked(). Done means the server reports success only after the file is synced and the final rename has completed, with downloads no longer returning incomplete data.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, rust
- Domain
- backend, build-system
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 58/100