dandi / dandi/dandi-archive

"Smart"/dynamic default page size-ing?

Open
#1,906 0 comments 0 reactions 0 assignees View on GitHub
performance
Dominant language
Python
Stars
26
Forks
21
Avg merge
4d 23h
Merged PRs (30d)
15

Description

Make paging smarter. Might be a little non-trivial to implement on django side (although didn't research -- might have been already done etc), but I think we might benefit from having first pages to be small (e.g. default 100) but then if request is going through subsequent pages -- make them as large as we can afford (or just 1000).
This way overall we can decrease desired number of requests for e.g. listing `/assets` in a dandiset 10-fold while consuming the same bandwidth (well -- actually even save that a little but not significantly).

For client code I believe it should not matter how many are actually returned in the response if no explicit page_size was requested.

ref:
- https://github.com/dandi/dandi-archive/issues/1891

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the paging behavior for the /assets endpoint in the Django API and the discussion in issue #1891. Determine how requests distinguish an explicit page_size from the default, then assess whether later pages can safely use a larger default without changing client behavior. Done means subsequent unconfigured pages return more results while explicit page_size requests remain unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
django, python
Domain
api, backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.