Mechanism to limit global memory usage
Open
- Dominant language
- C++
- Stars
- 1.5k
- Forks
- 145
- PR merge metrics
- No merged PRs in 30d
Description
If there a method to limit global memory usage?
Say I want to do something simple such as copying one array to another. Implementing this as follows can consume a lot of memory if the arrays are large?
```
import tensorstore as ts
input_arr = ts.open({ ... })
output_arr = ts.open({ ... })
output_arr.write(input_array).result()
```
I usually end up having to manually loop over the arrays chunk by chunk to force it to copy in parts. I'm aware of the `Context` API and concurrency limits, but this does not seem to have effectively constrain operations such as the one above.
Is there some way to limit the total memory used to avoid the Out Of Memory killer?
Contributor guide
Assessment
This issue has not been assessed yet.