owncloud / owncloud/ocis

pagination of large file listings, shares or trash

Open
#4,542 0 comments 0 reactions 0 assignees View on GitHub
Category:Enhancement
Dominant language
Go
Stars
2.1k
Forks
274
Avg merge
2d 1h
Merged PRs (30d)
103

Description

Currently, listing large folders (10k+) becomes slow when expensive properties are requested. Not only because the server may internally have to make more requests, but also because parsing XML with 10k+ entries and handling them in browsers becomes challenging.

# Add WebDAV pagination
We could use the [`Range` header](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Range) to let clients request a certain range of collections, eg. `Range: rows=2-3` and in the response return [`Content-Range: rows 2-3; total=9`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Range). And we can announce range suppurt using [`Accept-Ranges: rows`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept-Ranges) It would be compatible with the webdav RFCs, as long as we assume collections are ordered. The problem is that the web ui allows ordering by different properties and there is no example for how to specify the order of a collection in a PROPFIND request.

> Note [Web Distributed Authoring and Versioning (WebDAV) Ordered Collections Protocol
RFC 3648](https://datatracker.ietf.org/doc/rfc3648/) treats order as a property of the collection that is set at creation or PATCHed later. It does not define any semantics that could be (mis)used to make the server list resources in a specific order.

So we would have to introduce `OC-OrderBy: ` to let clients tell the server how to order the list.

# Use /graph/ driveItems

It already has [`$top`](https://docs.oasis-open.org/odata/odata/v4.01/odata-v4.01-part1-protocol.html#sec_SystemQueryOptiontop) & [`$skip`](https://docs.oasis-open.org/odata/odata/v4.01/odata-v4.01-part1-protocol.html#sec_SystemQueryOptionskip) for client driven pagination, [server driven pagination](https://docs.oasis-open.org/odata/odata/v4.01/odata-v4.01-part1-protocol.html#sec_ServerDrivenPaging) and [`$orderby`](https://docs.oasis-open.org/odata/odata/v4.01/odata-v4.01-part1-protocol.html#sec_SystemQueryOptionorderby) specified. (as well as `$filter`ing and `$expand`ing of resources)

# common consequences
Regardless of the approach, sorting would have to happen on the server side, which would make the ocdav or graph services consume a lot more memory. However, we can then use ListContainer together with an `If-Match` etag and reduce number of requests to the storage.

# related problems
- slow listing of shares: https://github.com/owncloud/ocis/issues/9828
- listing trash becomes slow with lots of items

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.