ga4gh / ga4gh/data-connect

Specify an optional page index to help clients get data more efficiently

Open
#109 1 comment 0 reactions 0 assignees View on GitHub
feature
Dominant language
HTML
Stars
25
Forks
14
PR merge metrics
No merged PRs in 30d

Description

This issue was raised during discussion on #98.

The goals of this **optional** "page index" would be:

1. tell consumers what partitions exist in the data, so they can skip partitions they don't care about
2. tell consumers which attributes the data is already sorted by
3. allow consumers to jump directly to the pages of interest
4. allow consumers to pull data in parallel (the strict singly-linked pagination structure in Search 1.0 is not amenable to consuming pages in parallel)
5. Fully optional for data producers/publishers (a page index is optional, not required)
6. Fully optional for data consumers (even if a page index is present, the consumer can still encounter all the data in order by following the singly-linked `next_page_url` sequence)

**Background**

We've experimented with something very basic along these lines with [the index attribute here](https://search-presto-public.prod.dnastack.com/tables). This index (a nonstandard experimental extension to Search) allows consumers to jump directly to a catalog of interest without having to page through all preceding catalogs in the sequence.

Imagine something like this on a Table object, which advertises information about partitions and sortedness already built into the data:

```json
"index": {
"ordered_by": [ "id" ],
"pages": [
{
"url": "https://storage.googleapis.com/ga4gh-phenopackets-example/flat/table/hpo_phenopackets/data_1",
"partitions": {
"id": {
"min": "PMID:27435956-Naz_Villalba-2016-NLRP3-proband",
"max": "PMID:29174093-Szczałuba-2018-GNB1-proband"
}
}
},
{
"url": "https://storage.googleapis.com/ga4gh-phenopackets-example/flat/table/hpo_phenopackets/data_2",
"partitions": {
"id": {
"min": "PMID:26833330-Jansen-2016-TMEM199-F1-II2",
"max": "PMID:27974811-Haliloglu-2017-PIEZO2-Patient"
}
}
}
]
}
}
```

This would indicate the data is partitioned by id and that the data consumer could skip partitions it does not need, and consume those that it does need in parallel.

This could be extended to partitions across multiple attributes, nested attributes, and even multiple available sort orders. We will have to balance complexity against performance benefits. Thanks to @ifokkema for initial feedback!

The above is just a rough illustration of the idea; don't take the exact format too seriously.

Contributor guide

Open the contributing guide

Research direction

Start by reading the discussion on issue #98 and the Search 1.0 pagination model, then examine the Table object and the experimental index attribute described here. Done means agreeing on an optional page-index representation, its partition and ordering semantics, and how it preserves sequential consumption while enabling parallel access.

Written by the indexing model from the issue text.

Assessment

Tech stack
json
Domain
api, data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.