Blosc / Blosc/python-blosc2

Issues with user-defined codecs

Open
#108 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
211
Forks
58
Avg merge
1d 17h
Merged PRs (30d)
6

Description

I'm facing issues when creating a simple codec that just makes a copy of the data to get familiar with Blosc's registering machinery. I attach the code:

```python
import blosc2
import numpy as np

# Create an User-defined codec (just a memcpy)

def encoder(input, output, meta, schunk: blosc2.SChunk):
print(f"Encoder output size: {output.size}")
output[:schunk.blocksize] = input[:schunk.blocksize]
return schunk.blocksize

def decoder(input, output, meta, schunk: blosc2.SChunk):
output[:schunk.blocksize] = input[:schunk.blocksize]
return schunk.blocksize

# Register the codec
codec_id = 200
blosc2.register_codec('test1', codec_id, encoder, decoder)

# Compress this array with the new codec

shape = (100, 100)
a = np.ones(shape, dtype=np.int64)

cparams = {
'codec': codec_id,
'nthreads': 1,
'filters': [],
'splitmode': blosc2.SplitMode.NEVER_SPLIT,
}

dparams = {
'nthreads': 1,
}

chunks = shape
blocks = (50, 50)

c_a = blosc2.asarray(a, chunks=chunks, blocks=blocks, cparams=cparams, dparams=dparams)
```

However, when I run the previous code, I get the following:
```
Encoder output size: 20000
Encoder output size: 20000
Encoder output size: 20000
Encoder output size: 19968

ValueError: could not broadcast input array from shape (20000,) into shape (19968,)
```

Looking at this, it appears that the last block of the chunk is smaller than the others. Do you know what is happening? Is there something I'm doing wrong?

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the supplied Python example and read the register_codec callbacks together with the asarray block configuration. Compare the input and output sizes for the final block, then determine how the codec should handle that partial block; done means the example completes without the broadcast error.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.