google / google/tensorstore

Mechanism to limit global memory usage

Open
#213 2 comments 1 reaction 0 assignees View on GitHub
Dominant language
C++
Stars
1.5k
Forks
145
PR merge metrics
No merged PRs in 30d

Description

If there a method to limit global memory usage?

Say I want to do something simple such as copying one array to another. Implementing this as follows can consume a lot of memory if the arrays are large?

```
import tensorstore as ts

input_arr = ts.open({ ... })
output_arr = ts.open({ ... })
output_arr.write(input_array).result()
```

I usually end up having to manually loop over the arrays chunk by chunk to force it to copy in parts. I'm aware of the `Context` API and concurrency limits, but this does not seem to have effectively constrain operations such as the one above.

Is there some way to limit the total memory used to avoid the Out Of Memory killer?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.