Materials-Consortia / Materials-Consortia/optimade-python-tools
Adding option to return binary data
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 91
- Forks
- 54
- PR merge metrics
- No merged PRs in 30d
Description
@giovannipizzi @ml-evs
For the upcoming trajectories endpoint, it would be nice to support a binary output format.
I made a small test script and managed to return the Cartesian site coordinates in the hdf5 format, which is much faster than with the JSON format. Although in the test case, there was no overhead from processing the query and such.
If we are using the JSON converter from FAST API, the JSON converter is called for each element of a list, which makes it quite slow. In addition, the numerical values have to be converted to strings, which is not necessary for the hdf5 format.
So I thought it would be good to implement the hdf5 format.
On closer inspection, I however noticed that the hdf5 format does not seem to support nested dictionaries like we use in the data.
A possible solution would be to use a dictionary instead of a list. For example, by replacing the indexes by a number or the id/name.
This would however be a deviation from the Optimade standard.
I was therefore wondering whether you think this is a problem, and if so, whether you know of other file formats that are able to efficiently store binary numbers while maintaining the Optimade data structure.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the upcoming trajectories endpoint and the FastAPI JSON conversion path described in the issue, then compare them with the proposed HDF5 representation. Done means choosing a binary format that preserves the OPTIMADE data structure and resolving whether implementation should proceed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- fastapi, python
- Domain
- api, backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100