HDFGroup / HDFGroup/hsds

Serving datasets with bitshuffle compression in POSIX files

Open
#76 34 comments 0 reactions 1 assignee Assigned to @mattjala View on GitHub
enhancement
Dominant language
Python
Stars
157
Forks
60
Avg merge
1d 5h
Merged PRs (30d)
15

Description

I am trying to serve POSIX files that contain datasets compressed with the [bitshuffle filter](https://github.com/kiyo-masui/bitshuffle).

`hsload --link` works without any trouble as do requests to metadata and uncompressed datasets. However, requests to the compressed datasets fail with the following errors in the datanode:
```
Error handling request
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/aiohttp/web_protocol.py", line 418, in start
resp = await task
File "/usr/local/lib/python3.8/site-packages/aiohttp/web_app.py", line 458, in _handle
resp = await handler(request)
File "/usr/local/lib/python3.8/site-packages/hsds/chunk_dn.py", line 259, in GET_Chunk
chunk_arr = await get_chunk(app, chunk_id, dset_json, bucket=bucket, s3path=s3path, s3offset=s3offset, s3size=s3size, chunk_init=False)
File "/usr/local/lib/python3.8/site-packages/hsds/datanode_lib.py", line 386, in get_chunk
chunk_arr = bytesToArray(chunk_bytes, dt, dims)
File "/usr/local/lib/python3.8/site-packages/hsds/util/arrayUtil.py", line 453, in bytesToArray
arr = np.frombuffer(data, dtype=dt)
ValueError: buffer size must be a multiple of element size
```

and

```
Error handling request
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/aiohttp/web_protocol.py", line 418, in start
resp = await task
File "/usr/local/lib/python3.8/site-packages/aiohttp/web_app.py", line 458, in _handle
resp = await handler(request)
File "/usr/local/lib/python3.8/site-packages/hsds/chunk_dn.py", line 259, in GET_Chunk
chunk_arr = await get_chunk(app, chunk_id, dset_json, bucket=bucket, s3path=s3path, s3offset=s3offset, s3size=s3size, chunk_init=False)
File "/usr/local/lib/python3.8/site-packages/hsds/datanode_lib.py", line 386, in get_chunk
chunk_arr = bytesToArray(chunk_bytes, dt, dims)
File "/usr/local/lib/python3.8/site-packages/hsds/util/arrayUtil.py", line 459, in bytesToArray
arr = arr.reshape(shape)
ValueError: cannot reshape array of size 6987 into shape (66,123)
```
.

How should I proceed to be able to request such datasets ? Given that I do `hsload --link`, should I look into HSDS rather than h5pyd ?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.