MemberJunction / MemberJunction/MJ

DropboxFileStorage.ListObjects does not paginate — large folders are silently truncated

Open
#4,439 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TSQL
Stars
29
Forks
6
Avg merge
2d 1h
Merged PRs (30d)
323

Description

## Problem

`DropboxFileStorage.ListObjects` calls `filesListFolder` once and never follows the cursor. Across all 1,741 lines of `packages/MJStorage/src/drivers/DropboxFileStorage.ts`, `filesListFolderContinue` appears **zero times**.

The pagination signals are read and then dropped:

```ts
// ListObjects, ~line 835
console.log('[DropboxFileStorage] filesListFolder response:', {
entriesCount: response.result.entries.length,
has_more: response.result.has_more, // logged
cursor: response.result.cursor, // logged
});
```

`SearchFiles` has the mirror image — it returns `nextPageToken: hasMore ? 'continue' : undefined` to its caller, but nothing in the driver ever calls the continue endpoint.

No explicit `limit` is set, so the request takes Dropbox's default page size. Any folder with more entries than that returns a truncated listing, and the caller cannot tell.

## Why it matters now

#4411 made `AutotagCloudStorage.ListModifiedObjects` walk the folder tree recursively instead of listing one level. That is the right fix, but it means many more folders get listed per run, so the chance that at least one exceeds a page rises with vault size. A Knowledge Hub crawl can report success having silently skipped files.

## Scope

Pre-existing driver behavior — not introduced by #4411. Filed as the follow-up agreed in that review.

## Suggested fix

Loop on `has_more` / `cursor` via `filesListFolderContinue` inside `ListObjects`, accumulating `entries` before converting to `objects` / `prefixes`. `SearchFiles` deserves the same treatment or an explicit doc note that the caller must page it.

Found during review of #4411.

Contributor guide

Open the contributing guide

Research direction

Start in packages/MJStorage/src/drivers/DropboxFileStorage.ts, reading ListObjects around line 835 and then SearchFiles. Trace the filesListFolder response fields and the filesListFolderContinue endpoint before changing the pagination behavior. Done means large folders are fully represented and SearchFiles either pages results too or clearly documents its caller's responsibility.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend, cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
75/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.