Serving datasets with bitshuffle compression in POSIX files
- Dominant language
- Python
- Stars
- 157
- Forks
- 60
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 15
Description
I am trying to serve POSIX files that contain datasets compressed with the [bitshuffle filter](https://github.com/kiyo-masui/bitshuffle).
`hsload --link` works without any trouble as do requests to metadata and uncompressed datasets. However, requests to the compressed datasets fail with the following errors in the datanode:
```
Error handling request
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/aiohttp/web_protocol.py", line 418, in start
resp = await task
File "/usr/local/lib/python3.8/site-packages/aiohttp/web_app.py", line 458, in _handle
resp = await handler(request)
File "/usr/local/lib/python3.8/site-packages/hsds/chunk_dn.py", line 259, in GET_Chunk
chunk_arr = await get_chunk(app, chunk_id, dset_json, bucket=bucket, s3path=s3path, s3offset=s3offset, s3size=s3size, chunk_init=False)
File "/usr/local/lib/python3.8/site-packages/hsds/datanode_lib.py", line 386, in get_chunk
chunk_arr = bytesToArray(chunk_bytes, dt, dims)
File "/usr/local/lib/python3.8/site-packages/hsds/util/arrayUtil.py", line 453, in bytesToArray
arr = np.frombuffer(data, dtype=dt)
ValueError: buffer size must be a multiple of element size
```
and
```
Error handling request
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/aiohttp/web_protocol.py", line 418, in start
resp = await task
File "/usr/local/lib/python3.8/site-packages/aiohttp/web_app.py", line 458, in _handle
resp = await handler(request)
File "/usr/local/lib/python3.8/site-packages/hsds/chunk_dn.py", line 259, in GET_Chunk
chunk_arr = await get_chunk(app, chunk_id, dset_json, bucket=bucket, s3path=s3path, s3offset=s3offset, s3size=s3size, chunk_init=False)
File "/usr/local/lib/python3.8/site-packages/hsds/datanode_lib.py", line 386, in get_chunk
chunk_arr = bytesToArray(chunk_bytes, dt, dims)
File "/usr/local/lib/python3.8/site-packages/hsds/util/arrayUtil.py", line 459, in bytesToArray
arr = arr.reshape(shape)
ValueError: cannot reshape array of size 6987 into shape (66,123)
```
.
How should I proceed to be able to request such datasets ? Given that I do `hsload --link`, should I look into HSDS rather than h5pyd ?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.