BlueBrain / BlueBrain/nexus

HTTP Range request on Files

Open
#2,948 0 comments 0 reactions 0 assignees View on GitHub
delta feature
Dominant language
Scala
Stars
301
Forks
76
PR merge metrics
No merged PRs in 30d

Description

In the context of data consumption from a client (frontend of a web app, in particular, the Atlas), it would be great to be able to fetch only sections of files using [http range requests](https://developer.mozilla.org/en-US/docs/Web/HTTP/Range_requests) (`Accept-Ranges: bytes`), and multipart ranges would be even better. There are two main use cases but the first is generic and is valid for many situations:

1. The container file specification we've designed (RandomAccessBuffer or RAB ) is used to store ephys traces. One trace is composed of multiple 1D signals and those can be many MB large. Being able to fetch one of the signals in a random access fashion is made possible by the specifications of the RAB format but not by Delta. This would speed up the data fetching and allow the user to select only a subset of the data and reduce the wait time. RAB format will probably be used for other modalities than ephys in a near future.

2. Volumetric files (3D rasters, NRRD) can be fairly large when encoding floating-point values such as gene expressions or cell densities. So large that it is unthinkable to ask an Atlas user to wait until it is downloaded (plus this would bloat the internal memory of the browser). The NRRD format specifications make it possible for such file to not use internal compression and be fully random-accessible. By allowing range requests to Delta Files, we open the possibility to perform a fast 3D spatial lookup into a volume. For example, get the cell density of a given cell type at a specific (x, y, z).

Both those situations were tested on a range-request-allowed personal S3 bucket and worked well.

As far as I know, there is not really any workaround possible to achieve that (unless duplicating the data on another server and maintaining a cache), so this feature is quite impactful on the DKE side.

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by locating the Delta Files HTTP retrieval path and its existing file-serving tests, then compare current responses with the HTTP Range specification and the S3 behavior described here. Done means clients can retrieve byte ranges, including multipart ranges if supported, without downloading the entire file.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
api, backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.