ga4gh / ga4gh/data-connect

Set rules for gzip content encoding that's friendly to tables-in-a-bucket

Open
#108 1 comment 0 reactions 0 assignees View on GitHub
feature
Dominant language
HTML
Stars
25
Forks
14
PR merge metrics
No merged PRs in 30d

Description

This issue was raised in a discussion on #98.

When data is stored as (potentially paginated) Search Tables, not only can the JSON files be gzip compressed at rest, but they can be served as-is with a `content-encoding: gzip` header. This adds no burden to the web server, and minimal burden to the client receiving the data.

The vast majority of real-world HTTP clients understand `content-encoding: gzip` and transparently decompress data as it is received. However, according to the HTTP specification, we should only serve gzip-compressed data if the client requests it with an `accept-encoding: gzip` header.

If we specify that Search clients MUST be capable of dealing with `content-encoding: gzip` and MUST send an `accept-encoding: gzip` header, this will make it easy for tables-in-a-bucket Search implementations to serve compressed responses, speeding up transfers and reducing storage costs.

Notes on cloud support:

* "raw" S3 can be set up to send gzip data regardless of `accept-encoding`: https://cri.dev/posts/2013-07-16-how-to-gzip-compression-of-css-and-js-files-on-s3-with-s3cmd/
* S3 with cloudfront is adaptive based on `accept-encoding` in request: https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/ServingCompressedFiles.html
* GCS is adaptive based on `accept-encoding` in request: https://cloud.google.com/storage/docs/transcoding
* Azure storage can be set up to send gzip data regardless of `accept-encoding`: https://github.com/stefangordon/azure-storage-gzip-encoding/blob/master/README.md

Contributor guide

Open the contributing guide

Research direction

Start by reading the discussion in issue #98 and the HTTP specification requirements described here, then compare the S3, CloudFront, GCS, and Azure notes. Done means the Search Tables rules clearly define client requirements for accept-encoding and content-encoding gzip, including how compressed responses should be served.

Written by the indexing model from the issue text.

Assessment

Domain
api, backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.