google / google/tensorstore

Are write operations with the zarr Driver guaranteed to be thread- and process-safe?

Open
#198 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
1.5k
Forks
145
PR merge metrics
No merged PRs in 30d

Description

Suppose I have an existing on-disk Zarr array. If I were to have two separate processes that:

1. Open this Zarr array via `tensorstore.open`
2. Write to separate regions that potentially share the same chunks within the Zarr array

Are these two write operations guaranteed to write correctly?

For example, suppose `my.zarr` has a chunk shape of (64,64,64).

- Process 1 writes to (0,0,0):(64,64,32)
- Process 2 writes to (0,0,32):(64,64,64)

```python
# Process 1
path = "path/to/my.zarr"
arr = ts.open(
{
"driver": "zarr",
"kvstore": {"driver": "file", "path": path},
},
open=True,
read=True,
write=True,
create=False,
).result()

arr[(0,0,0):(64,64,32)] = 100
```

```python
# Process 2
path = "path/to/my.zarr"
arr = ts.open(...) # Same as Process 1

arr[(0,0,32):(64,64,64)] = 200
```

The only mention I could find was in the homepage, [under the list of highlights](https://google.github.io/tensorstore/index.html#highlights).
> _Supports safe, efficient access from multiple processes and machines via optimistic concurrency._

And some basic testing seems to suggest that this is indeed true.

However, is this guaranteed to be the case? Is there anything within the documentation that provides this guarantee?

_P.S. Out of curiosity, how is the OCC actually implemented? Checking the last modified date of the Zarr chunk in which to write, or something along these lines?_
_P.P.S. Great library, by the way_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.