pagination of large file listings, shares or trash
- Dominant language
- Go
- Stars
- 2.1k
- Forks
- 274
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 103
Description
Currently, listing large folders (10k+) becomes slow when expensive properties are requested. Not only because the server may internally have to make more requests, but also because parsing XML with 10k+ entries and handling them in browsers becomes challenging.
# Add WebDAV pagination
We could use the [`Range` header](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Range) to let clients request a certain range of collections, eg. `Range: rows=2-3` and in the response return [`Content-Range: rows 2-3; total=9`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Range). And we can announce range suppurt using [`Accept-Ranges: rows`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept-Ranges) It would be compatible with the webdav RFCs, as long as we assume collections are ordered. The problem is that the web ui allows ordering by different properties and there is no example for how to specify the order of a collection in a PROPFIND request.
> Note [Web Distributed Authoring and Versioning (WebDAV) Ordered Collections Protocol
RFC 3648](https://datatracker.ietf.org/doc/rfc3648/) treats order as a property of the collection that is set at creation or PATCHed later. It does not define any semantics that could be (mis)used to make the server list resources in a specific order.
So we would have to introduce `OC-OrderBy: ` to let clients tell the server how to order the list.
# Use /graph/ driveItems
It already has [`$top`](https://docs.oasis-open.org/odata/odata/v4.01/odata-v4.01-part1-protocol.html#sec_SystemQueryOptiontop) & [`$skip`](https://docs.oasis-open.org/odata/odata/v4.01/odata-v4.01-part1-protocol.html#sec_SystemQueryOptionskip) for client driven pagination, [server driven pagination](https://docs.oasis-open.org/odata/odata/v4.01/odata-v4.01-part1-protocol.html#sec_ServerDrivenPaging) and [`$orderby`](https://docs.oasis-open.org/odata/odata/v4.01/odata-v4.01-part1-protocol.html#sec_SystemQueryOptionorderby) specified. (as well as `$filter`ing and `$expand`ing of resources)
# common consequences
Regardless of the approach, sorting would have to happen on the server side, which would make the ocdav or graph services consume a lot more memory. However, we can then use ListContainer together with an `If-Match` etag and reduce number of requests to the storage.
# related problems
- slow listing of shares: https://github.com/owncloud/ocis/issues/9828
- listing trash becomes slow with lots of items
Contributor guide
Assessment
This issue has not been assessed yet.