pagination: scans require an extra request to fetch an empty page of items

Currently, the next_page token on ResultsPage is populated whenever the page that we're returning to the client has any items in it. This works, but it means that for any scan through a collection, the client needs to make an extra request at the end to fetch an empty page in order to see that it's the end of the scan. Ideally, the last page with any results on it would indicate that there were no more results and we could skip this extra request. The problem is that right now, Dropshot doesn't know this information. One simple interface would be for the consumer to provide the limit that they used, and Dropshot could say that if they provided fewer items than the limit, then it's the end of the collection. (This would still require a theoretically request that's theoretically unnecessary if the scan ends with a full page of items, but this seems a lot less likely.) We could also ask the consumer to provide an explicit boolean indicating whether there are more items.

Another consideration: I've worked with systems that would fetched N records from the database, fail to process 1-2 of them, and send the results back to the client. If a consumer worked this way, using the interface I described above, we would erroneously conclude that because the page was non-full, we've finished the scan -- having omitted an arbitrarily large number of results after that. Obviously that's just a bug in the consumer, but it'd be nice to make that impossible if we can.

Start by reviewing the ResultsPage type and its next_page pagination contract. Compare the proposed limit-based and explicit-more-items interfaces, including consumers that omit records during processing. Done means the final non-empty page can reliably indicate whether more results exist without silently truncating scans.