Blosc / Blosc/python-blosc2

ZSTD compresion with dictionary causes odd errors

Open
#208 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
211
Forks
58
Avg merge
1d 17h
Merged PRs (30d)
6

Description

When activating the "use_dict" flag in an SChunk instance, storing data leads to errors.

The following code does not execute on my system:

```
import blosc2
import numpy as np

CHUNKSIZE = int(2**12)
NCHUNKS = 5

coptions = blosc2.cparams_dflts.copy()
coptions["codec"] = blosc2.Codec.ZSTD # this is already the default
coptions["use_dict"] = 1

_rng = np.random.default_rng()

def _make_data() -> bytes:
return _rng.random(CHUNKSIZE // 4, dtype=np.float32).tobytes()

data = [_make_data() for x in range(NCHUNKS)]

storage = blosc2.SChunk(
chunksize=CHUNKSIZE, cparams=coptions, dparams=blosc2.dparams_dflts
)

for x in data:
storage.append_data(x)

for index, x in enumerate(data):
assert storage.decompress_chunk(index) == x
```

Instead, it leads to the following `RuntimeError`:

```
Traceback (most recent call last):
File "/home/user/minimal_bug.py", line 26, in
storage.append_data(x)
File "/home/user/env/lib/python3.9/site-packages/blosc2/schunk.py", line 298, in append_data
return super(SChunk, self).append_data(data)
File "blosc2_ext.pyx", line 1105, in blosc2.blosc2_ext.SChunk.append_data
RuntimeError: Could not append the buffer
```

If the above code is run with `coptions["use_dict"] = 0`, it executes successfully.

Do specific flags need to be set for shared dictionary compression to be successful, or does the sizing of stored data have different requirements?

python-blosc2 version: `blosc2==2.3.2`
python version: `3.9.18`
platform: arch linux, conda based python install

Contributor guide

Open the contributing guide

Research direction

Reproduce the provided minimal script with python-blosc2 2.3.2 and inspect SChunk.append_data in schunk.py together with the blosc2_ext.pyx traceback path. Determine why use_dict=1 rejects these chunks, then verify that appending all five buffers and decompressing each one succeeds with the dictionary enabled.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.