Are write operations with the zarr Driver guaranteed to be thread- and process-safe?
- Dominant language
- C++
- Stars
- 1.5k
- Forks
- 145
- PR merge metrics
- No merged PRs in 30d
Description
Suppose I have an existing on-disk Zarr array. If I were to have two separate processes that:
1. Open this Zarr array via `tensorstore.open`
2. Write to separate regions that potentially share the same chunks within the Zarr array
Are these two write operations guaranteed to write correctly?
For example, suppose `my.zarr` has a chunk shape of (64,64,64).
- Process 1 writes to (0,0,0):(64,64,32)
- Process 2 writes to (0,0,32):(64,64,64)
```python
# Process 1
path = "path/to/my.zarr"
arr = ts.open(
{
"driver": "zarr",
"kvstore": {"driver": "file", "path": path},
},
open=True,
read=True,
write=True,
create=False,
).result()
arr[(0,0,0):(64,64,32)] = 100
```
```python
# Process 2
path = "path/to/my.zarr"
arr = ts.open(...) # Same as Process 1
arr[(0,0,32):(64,64,64)] = 200
```
The only mention I could find was in the homepage, [under the list of highlights](https://google.github.io/tensorstore/index.html#highlights).
> _Supports safe, efficient access from multiple processes and machines via optimistic concurrency._
And some basic testing seems to suggest that this is indeed true.
However, is this guaranteed to be the case? Is there anything within the documentation that provides this guarantee?
_P.S. Out of curiosity, how is the OCC actually implemented? Checking the last modified date of the Zarr chunk in which to write, or something along these lines?_
_P.P.S. Great library, by the way_
Contributor guide
Assessment
This issue has not been assessed yet.