FNNDSC / FNNDSC/fnndsc

Use iterators and discourage explicit pagination

Open
#100 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
7
Forks
5
PR merge metrics
No merged PRs in 30d

Description

Getters accept `limit` and `offset` parameters, which does not encourage the client to implement API pagination properly. For example, in the codebase for ChRIS_ui and chris_ui_ssr we see in many places where `{limit: 100*}` is given as a parameter.

```
$ grep -Fnr 'limit: 100' ChRIS_ui/src chris_ui_ssr/src
ChRIS_ui/src/api/common/index.ts:203: limit: 100,
ChRIS_ui/src/api/common/index.ts:220: limit: 1000,
ChRIS_ui/src/components/catalog/PluginCatalog.tsx:54: const plugins = await plugin.getPlugins({ limit: 1000 });
ChRIS_ui/src/components/feed/AddNode/GuidedConfig.tsx:63: limit: 1000,
ChRIS_ui/src/components/feed/AddPipeline/AddPipeline.tsx:82: limit: 1000,
ChRIS_ui/src/components/feed/AddTsNode/index.tsx:33: limit: 1000,
ChRIS_ui/src/components/feed/CreateFeed/utils/pipelines.ts:8: limit: 100,
ChRIS_ui/src/components/feed/CreateFeed/utils/pipelines.ts:25: limit: 1000,
ChRIS_ui/src/components/feed/NodeDetails/NodeDetails.tsx:61: limit: 100,
ChRIS_ui/src/components/feed/NodeDetails/NodeDetails.tsx:67: limit: 100,
ChRIS_ui/src/components/feed/Pipelines/GeneralCompute.tsx:25: limit: 100,
ChRIS_ui/src/components/feed/Pipelines/ConfigurationPage.tsx:50: limit: 1000,
ChRIS_ui/src/pages/DataLibrary/components/UserLibrary/BrowserContainer.tsx:122: limit: 100,
ChRIS_ui/src/pages/DataLibrary/components/UserLibrary/index.tsx:144: limit: 1000,
ChRIS_ui/src/pages/DataLibrary/components/UserLibrary/index.tsx:268: limit: 1000,
ChRIS_ui/src/pages/SinglePluginPage/SinglePlugin.tsx:46: limit: 1000,
ChRIS_ui/src/store/resources/saga.ts:42: const params = { limit: 100, offset: 0 };
chris_ui_ssr/src/lib/utilities/library/index.ts:44: limit: 10000000,
chris_ui_ssr/src/routes/(_main)/data/[...slug]/+page.server.ts:37: limit: 1000000,
chris_ui_ssr/src/routes/api/downloads/+server.ts:25: limit: 100000
chris_ui_ssr/src/routes/api/metadata/+server.ts:19: const files = await pathList.getFiles({ limit: 100000 });
```

This practice causes performance problems and code quality issues (pagination parameters are often specified in the code where pagination is not relevant in the context)

My suggestion is to provide an iterator (async iterable, stream, whatever it's called) based interface similar to how it is implemented in aiochris and the rust chris client.

https://fnndsc.github.io/aiochris/v0.4.0/aiochris.html#Search

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the listed limit call sites in ChRIS_ui/src and chris_ui_ssr/src, using the provided grep command to review how getters are consumed. Read the aiochris Search documentation and compare it with the rust client approach mentioned in the issue. Done would mean an agreed iterator-based interface that addresses these pagination usages without relying on arbitrary limits.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, typescript
Domain
api, frontend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.