openPMD / openPMD/openPMD-api

Simple slicing for chunked data

Open
#1,617 1 comment 0 reactions 0 assignees View on GitHub
question
Dominant language
C++
Stars
161
Forks
59
Avg merge
2d 22h
Merged PRs (30d)
7

Description

I just learned the long way that my naive approach to slice data (or even read full datasets) like
```python
mydata = it.meshes['cons_cons_density_lvl1'][opmd.Record_Component.SCALAR][:,:,10]
series.flush()
do_work_with_mydata()
```

Does not play nicely with chunked data (ADIOS2/mp5 output).

I saw the the openpmd-viewer introduces quite a bit of logic to load slice data https://github.com/openPMD/openPMD-viewer/blob/6eccb608893d2c9b8d158d950c3f0451898a80f6/openpmd_viewer/openpmd_timeseries/data_reader/io_reader/utilities.py#L88

Is there a simpler approach?
I naively would have expected that there's sth like `load_chunk*s*` (potentially hidden inside the `load_chunk` call) that would load the data from all chunks that are part of the dataset.

A related question to the approach above also pertains to performance: The viewer example load each chunk individually by calling many flushes. Is there a better(transparent way) around it (again ideally hidden behind a `load_chunk*s*` call)?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.