Materials-Consortia / Materials-Consortia/optimade-python-tools

Adding option to return binary data

Open
#1,285 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
91
Forks
54
PR merge metrics
No merged PRs in 30d

Description

@giovannipizzi @ml-evs

For the upcoming trajectories endpoint, it would be nice to support a binary output format.

I made a small test script and managed to return the Cartesian site coordinates in the hdf5 format, which is much faster than with the JSON format. Although in the test case, there was no overhead from processing the query and such.
If we are using the JSON converter from FAST API, the JSON converter is called for each element of a list, which makes it quite slow. In addition, the numerical values have to be converted to strings, which is not necessary for the hdf5 format.

So I thought it would be good to implement the hdf5 format.
On closer inspection, I however noticed that the hdf5 format does not seem to support nested dictionaries like we use in the data.
A possible solution would be to use a dictionary instead of a list. For example, by replacing the indexes by a number or the id/name.
This would however be a deviation from the Optimade standard.

I was therefore wondering whether you think this is a problem, and if so, whether you know of other file formats that are able to efficiently store binary numbers while maintaining the Optimade data structure.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the upcoming trajectories endpoint and the FastAPI JSON conversion path described in the issue, then compare them with the proposed HDF5 representation. Done means choosing a binary format that preserves the OPTIMADE data structure and resolving whether implementation should proceed.

Written by the indexing model from the issue text.

Assessment

Tech stack
fastapi, python
Domain
api, backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.