google / google/tensorstore

distributed appending? (like `zarr.append()`)

Open
#72 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
1.5k
Forks
145
PR merge metrics
No merged PRs in 30d

Description

Hello. First I'd like to refer to my previous issue #67, which explains my use case.

I want to **append** embeddings to tensorstore from multiple processes of pods. There is `append()` in `zarr`, but I couldn't find a equivalent function in tensorstore. How can I achieve the similar in tensorstore? below is my writing code. Thanks!

```python
def write_on_batch_end(
self,
trainer,
pl_module,
prediction: Any,
batch_indices: List[int],
batch: Any,
batch_idx: int,
dataloader_idx: int,
):
embed = prediction

# append to storage
# this function is called from multiple processes
...
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.