Windows-safe atomic writes for renameio
- Dominant language
- Go
- Stars
- 679
- Forks
- 34
- PR merge metrics
- No merged PRs in 30d
Description
This conversation started as an email exchange, and Michael suggested it should be made an issue, and I agreed with the suggestion. Therefore here is the full email exchange leading to creation of this issue:
NEIL
```
Hi Michael,
I'm Neil, a developer working on distributed systems, such as k3s, with a focus on crash-safety. I came across renameio via a k3s PR (k3s-io/k3s#14342) where I am using it to fix a bug. Well, the maintainer Brandon is hell bent on not having the code in the k3s, he was using large letters for "WE DO NOT", perhaps he is serious, or he doesn't like to be called out on bad code? I don't know.
The gap I noticed: renameio has no real atomic path on Windows. The workaround people are shipping (see rancher/dynamiclistener#313) is falling back to plain os.WriteFile, which loses the crash safety guarantee entirely on windows.
I'd like to implement a proper Windows-safe version (MoveFileEx / ReplaceFile with write-through) and take on maintaining it going forward, I know that's exactly the kind of surface area you may not want to own, but I don't mind owning it.
Would you be open to a PR along these lines?
Best,
Neil
```
MICHAEL
```
Hey Neil
Please see https://github.com/google/renameio/pull/5 for previous discussion.
The problem is that Windows does not support atomic writes, not even with MoveFileEx.
Are you saying that’s wrong? If so, which of the posts referenced from renameio#5 is wrong, and how?
Best regards
Michael
```
NEIL
```
Got it. Let me rebut you.
I believe when we say atomic replacement, we almost exclusively refer to the replacement operation itself, at least in the context of renameio.
It's just that if the contract we are discussing is "Readers observe either the old file or the new file, never the state in the middle," then I would say that yes, windows can provide those semantics on nfts. That's why windows applications and databases rely on it.
replacefile is specifically intended for atomically replacing an existing file.
BUT
If the contract is:
"The operating system explicitly guarantees semantics equivalent to posix across supported filesystems and scenarios."
Then the answer is not so clear.
But ultimately, there is one thing that I think stands out.
Requiring Windows to behave like posix is an impossible standard. If the criterion for support is "identical documented semantics to posix," then it will never be supported, regardless of what advancements it has or how good its native APIs become. Because it's not posix, it's windows, and it takes a very different approach to many areas.
A cross-platform library should expose the strongest native implementation available on each platform and document any semantic differences clearly.
Where it stands right now, Brandon, the maintainer of k3s, decided that it was okay to fall back to os writefile, an operation that can leave the destination truncated or partially written after a crash or power loss.
It's either we use the best implementation available for each platform, or people will fall back to or implement weaker alternatives with worse crash-safety guarantees.
In other words, either we expose the best implementation each platform provides, or we encourage downstream projects to ship software with weaker crash-safety guarantees than the platform could otherwise provide.
```
MICHAEL:
```
Have you seen https://pkg.go.dev/github.com/google/renameio/v2@v2.0.2/maybe?
If that’s not sufficient for your needs, can you open a GitHub issue for the discussion please?
This should be discussed in the open (so that others can participate), not via direct email.
```
Contributor guide
Research direction
Start by reading the discussion in renameio#47, the referenced pull request #5, and the package documentation for github.com/google/renameio/v2@v2.0.2/maybe. Before changing code, establish the supported Windows replacement and crash-safety semantics with the maintainers; done would require an agreed implementation and clearly documented platform differences.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- operating-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100