openPMD / openPMD/openPMD-api

Pickle: Mesh points at wrong series after deserializing

Open
#1,919 6 comments 0 reactions 0 assignees View on GitHub
affects latest release bug frontend: Python3
Dominant language
C++
Stars
161
Forks
59
Avg merge
2d 22h
Merged PRs (30d)
7

Description

**Describe the bug**
When serializing and deserializing a series object in python and doing the same for a `MeshRecord` (or `MeshRecordComponent` ?) both will successfully deserialize. But, although calling `load_chunk` on the deserialized mesh will work, calling `flush` on the deserialized series won't actually load the data.
There is no proper error message, but the buffer will contain random data. When serializing with some parallelization libraries like `joblib` workers can fail with messages like:
```
Fatal Python error: gilstate_tss_set: failed to set current tstate (TSS)
Python runtime state: finalizing (tstate=0x00000000017f9e18)
```

**To Reproduce**
`uv run deserialize_bug.py`
[deserialize_bug.py](https://github.com/user-attachments/files/30080371/deserialize_bug.py)

**Expected behavior**
Since the `MeshRecordComponent` and `MeshRecord` both have `add_pickle` both methods in the example should work

**Software Environment**
See attached file

Contributor guide

Open the contributing guide

Research direction

Start by running the attached deserialize_bug.py with `uv run deserialize_bug.py` and compare the deserialized MeshRecord or MeshRecordComponent path with the series path. Trace how `load_chunk` and `flush` behave after deserialization, including the `add_pickle` methods. Done means both examples load the expected data without random buffer contents or worker failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
api, data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.