HDFGroup / HDFGroup/hdf5-json

RFC: Dataset storage information in HDF5/JSON

Open
#75 6 comments 0 reactions 1 assignee Claimed by @ajelenak View on GitHub
Dominant language
Python
Stars
81
Forks
25
Avg merge
4h 46m
Merged PRs (30d)
5

Description

This is a proposal to add dataset storage information to HDF5/JSON. JSON key for this is named `byteBlocks`. The word "block" is hopefully still technically accurate while not too similar to "chunk".

Below is an example for one dataset. Block JSON keys are in the same format as in the HSDS schema, e.g. `0_0` and `0_1`. Two blocks are in the example; the `0_1` block has an optional `url` key in case of remote blocks. The `url` key can also apply to the entire dataset, in which case cannot appear in the blocks.

```json
{
"datasets": {
"7f335a2e-7ab1-11e4-87a5-3c15c2da029e": {
"attributes": [],
"dcpl": {
"fillValue": 0,
"layout": {
"class": "H5D_CHUNKED",
"dims": [8]
}
},
"shape": {
"class": "H5S_SIMPLE",
"dims": [10, 10],
"maxdims": [10, 10]
},
"type": {
"base": "H5T_STD_I32BE",
"class": "H5T_INTEGER"
},
"byteBlocks": {
"0_0": {
"offset": 1234,
"size": 2567,
},
"0_1": {
"offset": 56789,
"size": 1967,
"url": "s3://mybucket/path/to/object"
}
}
}
}
}
```

cc: @derobins @jreadey @gheber

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.