Specify an optional page index to help clients get data more efficiently
- Dominant language
- HTML
- Stars
- 25
- Forks
- 14
- PR merge metrics
- No merged PRs in 30d
Description
This issue was raised during discussion on #98.
The goals of this **optional** "page index" would be:
1. tell consumers what partitions exist in the data, so they can skip partitions they don't care about
2. tell consumers which attributes the data is already sorted by
3. allow consumers to jump directly to the pages of interest
4. allow consumers to pull data in parallel (the strict singly-linked pagination structure in Search 1.0 is not amenable to consuming pages in parallel)
5. Fully optional for data producers/publishers (a page index is optional, not required)
6. Fully optional for data consumers (even if a page index is present, the consumer can still encounter all the data in order by following the singly-linked `next_page_url` sequence)
**Background**
We've experimented with something very basic along these lines with [the index attribute here](https://search-presto-public.prod.dnastack.com/tables). This index (a nonstandard experimental extension to Search) allows consumers to jump directly to a catalog of interest without having to page through all preceding catalogs in the sequence.
Imagine something like this on a Table object, which advertises information about partitions and sortedness already built into the data:
```json
"index": {
"ordered_by": [ "id" ],
"pages": [
{
"url": "https://storage.googleapis.com/ga4gh-phenopackets-example/flat/table/hpo_phenopackets/data_1",
"partitions": {
"id": {
"min": "PMID:27435956-Naz_Villalba-2016-NLRP3-proband",
"max": "PMID:29174093-Szczałuba-2018-GNB1-proband"
}
}
},
{
"url": "https://storage.googleapis.com/ga4gh-phenopackets-example/flat/table/hpo_phenopackets/data_2",
"partitions": {
"id": {
"min": "PMID:26833330-Jansen-2016-TMEM199-F1-II2",
"max": "PMID:27974811-Haliloglu-2017-PIEZO2-Patient"
}
}
}
]
}
}
```
This would indicate the data is partitioned by id and that the data consumer could skip partitions it does not need, and consume those that it does need in parallel.
This could be extended to partitions across multiple attributes, nested attributes, and even multiple available sort orders. We will have to balance complexity against performance benefits. Thanks to @ifokkema for initial feedback!
The above is just a rough illustration of the idea; don't take the exact format too seriously.
Contributor guide
Research direction
Start by reading the discussion on issue #98 and the Search 1.0 pagination model, then examine the Table object and the experimental index attribute described here. Done means agreeing on an optional page-index representation, its partition and ordering semantics, and how it preserves sequential consumption while enabling parallel access.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- json
- Domain
- api, data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100