tee-ar-ex / tee-ar-ex/trx-python
NPY file format for regular matrix data
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 26
- Forks
- 20
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 9
Description
This idea was going to result in #15 getting peppered with repetitive comments, so I'm going to write it here separately instead.
TRX currently has novel handling of matrix dimensions & datatype for various data files, achieved via file names. When looking through the code in #15 I also see what looks like novel enumeration / single-character encoding of data type. This may be creating a novel solution for a problem for which many solutions already exist.
The NPY format provides an established solution for these issues. Matrix dimensions and data type (including endianness) are encoded in the file header as part of a dictionary literal. I've myself recently implemented C++ support for that format in https://github.com/MRtrix3/mrtrix3/pull/2437. Using this file format as part of the higher-order TRX format would be fairly trivial for Python, in particular facilitating reading / writing of data with no dependence on TRX libraries, and for other languages the overhead would be no greater than that demanded by the current specification. Potential downsides are that features such as matrix dimensionality / size and data type would no longer be visible from a filesystem view (though they could be pretty easily seen just using head), and memory-mapping implementations would need to support loading from a non-zero offset into a file (which shouldn't be difficult, it's a common operation). But the upsides in terms of not reinventing the wheel may more than offset that.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No implementation file or test is named. Start by reviewing the matrix-dimension and datatype handling discussed in #15 and the linked NPY format specification. Done means deciding whether NPY should be used within the higher-order TRX format and specifying the compatibility and memory-mapping implications.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100