google / google/renameio

Windows-safe atomic writes for renameio

Open
#47 8 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
679
Forks
34
PR merge metrics
No merged PRs in 30d

Description

This conversation started as an email exchange, and Michael suggested it should be made an issue, and I agreed with the suggestion. Therefore here is the full email exchange leading to creation of this issue:

NEIL
```
Hi Michael,

I'm Neil, a developer working on distributed systems, such as k3s, with a focus on crash-safety. I came across renameio via a k3s PR (k3s-io/k3s#14342) where I am using it to fix a bug. Well, the maintainer Brandon is hell bent on not having the code in the k3s, he was using large letters for "WE DO NOT", perhaps he is serious, or he doesn't like to be called out on bad code? I don't know.

The gap I noticed: renameio has no real atomic path on Windows. The workaround people are shipping (see rancher/dynamiclistener#313) is falling back to plain os.WriteFile, which loses the crash safety guarantee entirely on windows.

I'd like to implement a proper Windows-safe version (MoveFileEx / ReplaceFile with write-through) and take on maintaining it going forward, I know that's exactly the kind of surface area you may not want to own, but I don't mind owning it.

Would you be open to a PR along these lines?

Best,
Neil
```

MICHAEL
```
Hey Neil

Please see https://github.com/google/renameio/pull/5 for previous discussion.

The problem is that Windows does not support atomic writes, not even with MoveFileEx.
Are you saying that’s wrong? If so, which of the posts referenced from renameio#5 is wrong, and how?

Best regards
Michael
```

NEIL
```
Got it. Let me rebut you.

I believe when we say atomic replacement, we almost exclusively refer to the replacement operation itself, at least in the context of renameio.

It's just that if the contract we are discussing is "Readers observe either the old file or the new file, never the state in the middle," then I would say that yes, windows can provide those semantics on nfts. That's why windows applications and databases rely on it.

replacefile is specifically intended for atomically replacing an existing file.

BUT

If the contract is:

"The operating system explicitly guarantees semantics equivalent to posix across supported filesystems and scenarios."

Then the answer is not so clear.

But ultimately, there is one thing that I think stands out.

Requiring Windows to behave like posix is an impossible standard. If the criterion for support is "identical documented semantics to posix," then it will never be supported, regardless of what advancements it has or how good its native APIs become. Because it's not posix, it's windows, and it takes a very different approach to many areas.

A cross-platform library should expose the strongest native implementation available on each platform and document any semantic differences clearly.

Where it stands right now, Brandon, the maintainer of k3s, decided that it was okay to fall back to os writefile, an operation that can leave the destination truncated or partially written after a crash or power loss.

It's either we use the best implementation available for each platform, or people will fall back to or implement weaker alternatives with worse crash-safety guarantees.

In other words, either we expose the best implementation each platform provides, or we encourage downstream projects to ship software with weaker crash-safety guarantees than the platform could otherwise provide.
```

MICHAEL:
```
Have you seen https://pkg.go.dev/github.com/google/renameio/v2@v2.0.2/maybe?

If that’s not sufficient for your needs, can you open a GitHub issue for the discussion please?
This should be discussed in the open (so that others can participate), not via direct email.
```

Contributor guide

Open the contributing guide

Research direction

Start by reading the discussion in renameio#47, the referenced pull request #5, and the package documentation for github.com/google/renameio/v2@v2.0.2/maybe. Before changing code, establish the supported Windows replacement and crash-safety semantics with the maintainers; done would require an agreed implementation and clearly documented platform differences.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
operating-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.