dotnet / dotnet/efcore

Holistic end-to-end pagination feature

Open
#33,160 17 comments 17 reactions 0 assignees View on GitHub
area-perf area-query consider-for-next-release needs-design
Dominant language
C#
Stars
14.8k
Forks
3.4k
PR merge metrics
PR metrics pending

Description

EF currently provides various primitives for performing pagination (Take/Skip as well as keyset pagination - [see docs](https://learn.microsoft.com/en-us/ef/core/querying/pagination)). However, we regularly see a need for a higher-level, more end-to-end solution that would provide easy, efficient pagination capabilities without having to manually work everything together each time.

The basic API could be something like a terminating ToPageAsync() call (proposed by @AndriySvyryd in #24513, as well as by @michaelstaib in his work on GraphQL EF integration). The query would return a page containing the rows in the page, an easy means for fetching the next/previous page, and some additional data (HasNextPage, HasPreviousPage, TotalCount).

Following is a summary of general notes/requirements (thanks @michaelstaib for the useful discussion!):

* Paginating queries frequently need to know whether there's a previous and next page (for UI purpoess). This can be done by selecting out an additional row on each side, or something nicer is probably possible with [window functions](https://github.com/dotnet/efcore/issues/12747).
* Similarly, it's frequently desired to know the total number of rows when performing a paginating query, without doing an additional database query just for that. Doing this efficiently depends on [window function support](https://github.com/dotnet/efcore/issues/12747).
* It should be possible to extract some sort of "cursor" or "pagination/continuation token", which would allow fetching the next (or previous page), with a repeated invocation of ToPageAsync(). The actual contents/format of the token should be opaque, and will vary across databases/techniques (Cosmos has a continuation token, keyset pagination has its keyset, offset pagination has the numeric position). It should ideally be easy to get a string representation (or similar) to transfer the token to the client for disconnected scenarios.
* For keyset pagination, it's necessary to specify the sorting keys. Instead of requiring these to be specified explicitly, an interesting idea would be for the ToPageAsync() operator to infer the keys by examining the IOrderedQueryable it's built on top of.
* Cosmos support for pagination is tracked by [#24513](https://github.com/dotnet/efcore/issues/24513#issuecomment-1836962812). We ideally would arrive at an abstract API shape which can work for both relational and Cosmos.
* Note that paging isn't only a terminating/top-level concern - we should support it in filtered includes as well, to allow paging over dependent collections (e.g. `context.Blogs.Include(b => b.Posts.OrderBy(...).ThenBy(...).Page(...)`). This has particular importance to GraphQL (/cc @michaelstaib).
* We'd need to think about whether the API buffers the entire page, or whether it allows streaming. Streaming is probably not very important (since we're paginating anyway), and may complicate the API (additional data like TotalCount/HasNextPage won't be available before reading the first row).

Prior art: [MR.EntityFrameworkCore.KeysetPagination](https://github.com/mrahhal/MR.EntityFrameworkCore.KeysetPagination), [Pagination.EntityFrameworkCore.Extensions](https://github.com/SitholeWB/Pagination.EntityFrameworkCore.Extensions)

/cc @michaelstaib @mrahhal @SitholeWB

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.