paritytech / paritytech/json-rpc-interface-spec

`chainHead_storage`: Support more resilient pagination of descendants?

Open
#85 9 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
No language data
Stars
37
Forks
6
PR merge metrics
No merged PRs in 30d

Description

In the legacy RPC methods, a user could call state_getKeysPaged with some prefix, and provide a starting key (and number of keys to fetch). They are able to then paginate over the child keys/values by tracking the last key that they have seen and using that to request the next batch of keys (and obtain values for them). This is resistant to network connection interruptions and such; they can always pick up from the last key they are interested in.

Using the new chainHead_storage API, a couple of things make iterating over descendent keys/values harder:

  • There is no way to just ask for descendent keys (I guess I'd ask for descendentHashes and then just ignore them and look at the keys, but this is less than ideal).
  • There is no way to pick up from where I left off in case the chainHead subscription I'm relying on stops for any reason (eg network interruption or pinned blocks getting too old).
Proposals

Being able to provide a startKey to the chainHead_storage items of type descendantsValues and descendantsHashes would probably go most of the way here (I'm not super sure if it's useful to be able to fetch keys without fetching values in the usual case of paginating over some descendent results).

I also wondered about supporting a new type like descendantsKeys in storage items. This would return only the keys, and no other values (so that you can eg download all of the keys and then ask for values later as a user paginates through some interface or whatever). If the startKey idea is acceptable then this would also allow a startKey to align with other descendants* types, and allow for reliable pagination/resuming of fetching keys as well as values or hashes.

What do you guys reckon?

/cc @josepot @tomaka

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the chainHead_storage item types descendantsValues and descendantsHashes described in the issue. Determine whether startKey, a descendantsKeys type, or both are required, then define pagination and resume behavior and update the interface specification accordingly.

Written by the indexing model from the issue text.

Assessment

Domain
api, backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.