Local filesystem writes can retain stale tail data when overwriting shorter objects
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 1.6k
- Forks
- 438
- PR merge metrics
- No merged PRs in 30d
Description
Problem
The Local FS storage backend opens existing objects with os.O_RDWR|os.O_CREATE but without os.O_TRUNC:
handle, err := os.OpenFile(o.path, os.O_RDWR|os.O_CREATE, 0o644)
fsObject.Put writes from offset zero, so overwriting an existing object with shorter content leaves the old tail in the file. The same behavior affects the uncompressed StoreFile path that uses getHandle(false).
Example:
existing object: abcdefgh
new object: xy
read result: xycdefgh
This makes the Local FS backend inconsistent with object-storage backends, where a put replaces the complete object. It can corrupt local snapshots, cache files, or metadata when an object is rewritten with a smaller payload.
Proposed fix
Use truncation for complete-object writes, either by adding os.O_TRUNC to the write handle or by explicitly truncating and resetting the offset before copying:
handle, err := os.OpenFile(o.path, os.O_RDWR|os.O_CREATE|os.O_TRUNC, 0o644)
if err != nil {
return nil, err
}
The truncating mode should only be used for complete replacement writes. Read, range-read, append, and multipart/random-offset paths should retain their existing semantics.
Tests
Please add two separate regression tests:
-
Putpath:- Write a longer payload to an object.
- Overwrite it with a shorter payload using
Put. - Verify
Sizeequals the new payload length. - Verify reading returns exactly the new payload, with no stale suffix.
-
Direct, uncompressed
StoreFilepath:- Store a longer source file into an object.
- Overwrite it with a shorter source file using uncompressed
StoreFile. - Verify
Sizeequals the new source file length. - Verify reading returns exactly the new source content, with no stale suffix.
The compressed StoreFile path is a separate implementation and should not be conflated with this regression.
Relevant code:
packages/shared/pkg/storage/storage_fs.gofsObject.PutfsObject.StoreFilefsObject.getHandle
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read packages/shared/pkg/storage/storage_fs.go, focusing on fsObject.Put, fsObject.StoreFile, and fsObject.getHandle, then run the storage package tests. Add separate regression coverage for shorter overwrites through Put and uncompressed StoreFile; done means Size and reads contain exactly the replacement payload, while compressed StoreFile and non-replacement paths retain their existing behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 86/100