TraceMachina / TraceMachina/nativelink

CAS uploads succeed but stored files are incomplete due to async rename race condition

Open
#2,242 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
1.6k
Forks
244
Avg merge
1d 16h
Merged PRs (30d)
54

Description

When using the LUCI Go CAS client (cas.exe) to archive .isolate files, all resources appear to be uploaded successfully to the CAS server. However, the uploaded files on the server side are incomplete (missing data).

After investigating, the issue seems to originate from the NativeLink CAS server implementation, specifically in:

nativelink-store/src/filesystem_store.rs

Observed flow:

update_oneshot()
→ write temp file ✅
→ sync ✅
→ drop file ✅
→ call emplace_file()
→ spawn background task (rename not finished yet ❗)
→ return OK 🚨

Meanwhile, on the client side:

cas.exe
→ starts downloading immediately
→ calls get_part_unchunked()

Problem:

There is a race condition between upload completion and file availability:

update_oneshot() returns OK before the file is fully moved (renamed) to its final location.
The emplace_file() function spawns a background task to perform the rename, but does not wait for it to complete.
The client begins downloading immediately after receiving success.
At this point, the file may still be in the temporary path or the rename operation is not finished.

Result:

Clients may read incomplete or partially written files from the CAS server.

Expected behavior:

The server should only return success after:

The file has been fully written,
Synced to disk,
And atomically moved (rename completed) to its final location.

Suggested fix:

Ensure the rename operation in emplace_file() is completed synchronously before returning success,
or
Introduce a mechanism to guarantee file visibility/consistency before allowing reads.

Environment:

LUCI Go CAS client (cas.exe)
NativeLink CAS server

Let me know if more logs or a minimal reproduction are needed.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read nativelink-store/src/filesystem_store.rs, starting with update_oneshot() and emplace_file(), then reproduce the upload and immediate download flow with the LUCI Go cas.exe client and get_part_unchunked(). Done means the server reports success only after the file is synced and the final rename has completed, with downloads no longer returning incomplete data.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, rust
Domain
backend, build-system
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.