HTTP Range request on Files
- Dominant language
- Scala
- Stars
- 301
- Forks
- 76
- PR merge metrics
- No merged PRs in 30d
Description
In the context of data consumption from a client (frontend of a web app, in particular, the Atlas), it would be great to be able to fetch only sections of files using [http range requests](https://developer.mozilla.org/en-US/docs/Web/HTTP/Range_requests) (`Accept-Ranges: bytes`), and multipart ranges would be even better. There are two main use cases but the first is generic and is valid for many situations:
1. The container file specification we've designed (RandomAccessBuffer or RAB ) is used to store ephys traces. One trace is composed of multiple 1D signals and those can be many MB large. Being able to fetch one of the signals in a random access fashion is made possible by the specifications of the RAB format but not by Delta. This would speed up the data fetching and allow the user to select only a subset of the data and reduce the wait time. RAB format will probably be used for other modalities than ephys in a near future.
2. Volumetric files (3D rasters, NRRD) can be fairly large when encoding floating-point values such as gene expressions or cell densities. So large that it is unthinkable to ask an Atlas user to wait until it is downloaded (plus this would bloat the internal memory of the browser). The NRRD format specifications make it possible for such file to not use internal compression and be fully random-accessible. By allowing range requests to Delta Files, we open the possibility to perform a fast 3D spatial lookup into a volume. For example, get the cell density of a given cell type at a specific (x, y, z).
Both those situations were tested on a range-request-allowed personal S3 bucket and worked well.
As far as I know, there is not really any workaround possible to achieve that (unless duplicating the data on another server and maintaining a cache), so this feature is quite impactful on the DKE side.
Contributor guide
Research direction
No files, tests, or entry points are named. Start by locating the Delta Files HTTP retrieval path and its existing file-serving tests, then compare current responses with the HTTP Range specification and the S3 behavior described here. Done means clients can retrieve byte ranges, including multipart ranges if supported, without downloading the entire file.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- api, backend
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100