google / google/tensorstore

Performance drop on 1D Zarr file

Open
#231 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
1.5k
Forks
145
PR merge metrics
No merged PRs in 30d

Description

There is a performance degradation when writing and reading into a zarr3 file with 1 dimension.
for example, when reading a [1024,1024,1024] dataset completely it takes 4.66201 sec, but reshaping it to a [32768,32768] dataset and then read it reduce the read time to 2.59770.

Most parameters remained the same, as number of chunkes. each dataset is chunked into 64 pieces.

this is the piece of code used to capture the read time in both situation:

```
static void main_read_full(const char* path,json_object *main_obj, int loadnumber )
{
MACSIO_TIMING_GroupMask_t main_read_full_grp = MACSIO_TIMING_GroupMask("main_read_full");
MACSIO_TIMING_TimerId_t main_read_full_tid;
double timer_dt;

::nlohmann::json json_spec = readjson(num_th,path);
main_read_full_tid = MT_StartTimer("open", main_read_full_grp, loadnumber);
auto result = tensorstore::Open(json_spec, context, tensorstore::OpenMode::open);
timer_dt = MT_StopTimer(main_read_full_tid);
auto store = std::move(result).value();
main_read_full_tid = MT_StartTimer("read_data", main_read_full_grp, loadnumber);
auto read_result = tensorstore::Read(
store).result();
timer_dt = MT_StopTimer(main_read_full_tid);
if (!read_result.ok()) {
std::cerr << "Failed to read data: " << read_result.status() << std::endl;
}
}

```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.