Pometry / Pometry/Raphtory

Collection-axis paging: columnar reads are unusable on a disable_lists server

Open
#2,732 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
642
Forks
70
Avg merge
2d 13h
Merged PRs (30d)
21

Description

The remote client's columnar collection reads all render as { list { … } }, so a server with concurrency.disable_lists set rejects them — and the client exposes no page alternative for collections. Verified end to end against a disable_lists server:

call result
len(rg.nodes) works (scalar count)
rg.nodes.properties.keys() works (registry-backed propertyKeys, not list)
rg.nodes.id blocked
rg.nodes.degree() blocked
rg.nodes.properties.get("score") blocked
list(rg.nodes) blocked

disable_lists exists precisely to stop unbounded reads, so this is a supported configuration the client cannot work against.

Three axes — do not confuse them

Paging shows up in three different places, and only one is implemented:

  1. Within one entitynode.history.page(3). Implemented (HistoryPage/SubPage).
  2. Across a collection's membershipnodes.page(100). Server has it (GqlNodes::page); the client does not expose it.
  3. A column across a collectionnodes.degree(), nodes.id. Renders as list { degree }; no paged form exists on either side.

The history paging code (axis 1) cannot be reused for axis 3: it pages inside a single entity's event list, whereas a columnar read spans entities. Axis 2 returns node handles, not columns, so exposing it does not fix axis 3 either.

Options

(a) Chunk client-side. Columnar terminals loop over pages internally. Cost: breaks the "one RPC per terminal" invariant that test_parity_rpc_counts.py::test_terminal_fires_exactly_one_rpc asserts across 258 cases; also picks a chunk size on the user's behalf and risks torn reads across pages on a mutating graph.

(b) Paged columnar fields server-side (recommended). Add paged variants of the columnar fields (e.g. pagedDegree(limit:, offset:, pageIndex:)) on the collection types, so each call stays one RPC and disable_lists servers work properly. Cost: ~15 fields × 5 collection types, plus a schema change.

(c) Expose nodes.page(...) only. Half a day, mirrors the existing history paging exactly, gives users on a hardened server a way to read entity-by-entity. Does not fix columnar reads; documents the limit instead.

Recommendation: (b) as the real fix; (c) is a reasonable stop-gap if someone needs a hardened server working before (b) lands.

Related
  • #2725 — nested edge collections (the nested axis of the same story)
  • #2726 — remote history paging iterator (axis 1)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the existing GqlNodes::page implementation and the HistoryPage/SubPage paging code, keeping the three paging axes separate. Review test_parity_rpc_counts.py::test_terminal_fires_exactly_one_rpc to understand the current RPC invariant. Done means columnar collection reads have a paged form that works with disable_lists without breaking the one-RPC terminal behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
api, database
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.