distributed appending? (like `zarr.append()`)
- Dominant language
- C++
- Stars
- 1.5k
- Forks
- 145
- PR merge metrics
- No merged PRs in 30d
Description
Hello. First I'd like to refer to my previous issue #67, which explains my use case.
I want to **append** embeddings to tensorstore from multiple processes of pods. There is `append()` in `zarr`, but I couldn't find a equivalent function in tensorstore. How can I achieve the similar in tensorstore? below is my writing code. Thanks!
```python
def write_on_batch_end(
self,
trainer,
pl_module,
prediction: Any,
batch_indices: List[int],
batch: Any,
batch_idx: int,
dataloader_idx: int,
):
embed = prediction
# append to storage
# this function is called from multiple processes
...
```
Contributor guide
Assessment
This issue has not been assessed yet.