Tracking: Pagination

Going through pages of results in the external API is not very good right now. The biggest problem is that you can only go forward. If you have a cursor pointing to a page, there is simply no way to access the previous page without starting over at the beginning of the collection and paging through. Details are in the Dropshot issue, but most likely we need to be able to ask for the page before a given cursor, which would require splitting the existing page_token param into two: starting_from (same behavior as current page_token) and ending_before.

Another problem is that we will always give a next_page cursor regardless of whether there are actually any more items. As discussed in the Dropshot issue, we need to give the consumer (Nexus) a way of telling Dropshot whether there should be a next_page param, and it would be up to Nexus to determine the criterion. A very simple one that would require changing very little about our code would be to only include next_page if the current page's size equals the current page size limit. This would have a 1/limit chance of being wrong, which is not that bad. A more robust but more complex approach would involve fetching limit + 1 results for a given page so we would actually know whether there are more items. The extra precision here is somewhat undermined by the fact that the actual number of items available could change between requests regardless.

This mostly depends on work that would take place in Dropshot, so fittingly both of these issues are captured there.

- [ ] https://github.com/oxidecomputer/dropshot/issues/436
- [ ] https://github.com/oxidecomputer/dropshot/issues/20
- [ ] https://github.com/oxidecomputer/omicron/issues/2343

Start by reading Dropshot issues #436 and #20, followed by Omicron issue #2343, since this work depends on changes outside this issue. Trace how those changes affect Omicron's external API pagination. Done means previous-page access is available and the API can avoid advertising next_page when no further results exist.