MemberJunction / MemberJunction/MJ
DropboxFileStorage.ListObjects does not paginate — large folders are silently truncated
- Dominant language
- TSQL
- Stars
- 29
- Forks
- 6
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 323
Description
## Problem
`DropboxFileStorage.ListObjects` calls `filesListFolder` once and never follows the cursor. Across all 1,741 lines of `packages/MJStorage/src/drivers/DropboxFileStorage.ts`, `filesListFolderContinue` appears **zero times**.
The pagination signals are read and then dropped:
```ts
// ListObjects, ~line 835
console.log('[DropboxFileStorage] filesListFolder response:', {
entriesCount: response.result.entries.length,
has_more: response.result.has_more, // logged
cursor: response.result.cursor, // logged
});
```
`SearchFiles` has the mirror image — it returns `nextPageToken: hasMore ? 'continue' : undefined` to its caller, but nothing in the driver ever calls the continue endpoint.
No explicit `limit` is set, so the request takes Dropbox's default page size. Any folder with more entries than that returns a truncated listing, and the caller cannot tell.
## Why it matters now
#4411 made `AutotagCloudStorage.ListModifiedObjects` walk the folder tree recursively instead of listing one level. That is the right fix, but it means many more folders get listed per run, so the chance that at least one exceeds a page rises with vault size. A Knowledge Hub crawl can report success having silently skipped files.
## Scope
Pre-existing driver behavior — not introduced by #4411. Filed as the follow-up agreed in that review.
## Suggested fix
Loop on `has_more` / `cursor` via `filesListFolderContinue` inside `ListObjects`, accumulating `entries` before converting to `objects` / `prefixes`. `SearchFiles` deserves the same treatment or an explicit doc note that the caller must page it.
Found during review of #4411.
Contributor guide
Research direction
Start in packages/MJStorage/src/drivers/DropboxFileStorage.ts, reading ListObjects around line 835 and then SearchFiles. Trace the filesListFolder response fields and the filesListFolderContinue endpoint before changing the pagination behavior. Done means large folders are fully represented and SearchFiles either pages results too or clearly documents its caller's responsibility.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 75/100