Blosc / Blosc/c-blosc2

Use msgpack for the chunks section in frame format

Open
#76 0 comments 0 reactions 1 assignee Claimed by @FrancescAlted View on GitHub
enhancement high priority
Dominant language
C
Stars
584
Forks
112
PR merge metrics
No merged PRs in 30d

Description

Packing the chunks section of the [frame format](https://github.com/Blosc/c-blosc2/tree/master/README_FRAME_FORMAT.rst) would bring more encapsulation and better maneagibility in the end. With that, fields like `uncompressed_size` and `compressed_size` and `blocksize` can be moved into the new `chunks` section.

Suggestion: add a `chunks_size` field to the new `chunks` so that computing the size of the chunk for offsets would be easier, and specially, faster (e.g. remove the need for `get_trailer_offsets()` in `frame_get_chunk()`).

After doing this, I am not sure on whether we should add another msgpck field at the front of the header for specifying that three other msgpack sections come later (header, chunks and trailer). Probably this is not a good idea because then utilities like `msgpack2json` may want to read the whole file, and this can be too much for large files.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.