ipld / ipld/go-car

blockstore: OpenReadWrite should use atomic writes

Open
#249 1 comment 0 reactions 0 assignees View on GitHub
help wanted P3
Dominant language
Go
Stars
175
Forks
54
Avg merge
6d 5h
Merged PRs (30d)
4

Description

Right now, OpenReadWrite writes directly to the destination file. This has multiple problems:

* If we encounter some error mid-finalize, we may leave a corrupted file
* If we encounter a write/flush error while finalizing, we may leave a partial file
* If we're resuming on an existing file and writing in-place, we may corrupt the user's data (e.g. https://github.com/ipld/go-car/issues/247)

It would be much saner overall to instead write to a temporary copy next to the destination file (such as `foo.car.tmp` for `foo.car`), and once `Finalize` has finished with no error, we do a rename that should basically never fail.

This also means that we can teach OpenReadWrite to always remove `foo.car.tmp` when it's discarded, to ensure we don't leave unfinished business behind.

The method described above should be doable with just a bit of `os` and `io` glue. We could use https://pkg.go.dev/github.com/google/renameio#TempFile, though I think pulling in a library for this use case is perhaps overkill. We want to prevent common errors, and we're not *that* concerned with atomicity between multiple processes - users should not be writing to the same CAR file concurrently.

cc @masih @willscott

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.