Oximeter collector API needs to be paginated

The current implementation of the Dropshot endpoint that the oximeter collector uses to scrape metric data is not paginated. It should be, for at least two reasons:

• The same reason we paginate any API, such as limiting memory consumption and avoiding denial-of-service type attacks.
• It provides a measure of robustness, making it less likely we'll lose data.

The former is self-explanatory, but the latter deserves some expansion. Right now, oximeter collects all available data from a metric producer. All consumers currently use the oximeter_producer::ProducerRegistry type to do this. When the server gets a request from oximeter, that type pulls all available data from its registered producers, and sends it over the network. At that point, it's gone from the producer.

oximeter does some work on the data, including figuring out which timeseries they belong to, constructing the database records, and inserting them in batches. Should oximeter crash between when the data leaves the producer and the time its inserted into the database, it will be gone for good. Paginating could help prevent this, if it's designed correctly.

The idea (credit to @ahl) would be to have the oximeter collect a limited set of samples. The producer packages them up and sends them in the response, but does not drop them. They are moved to a holding area, along with the page token (or some other identifier) that's included in the response sent to oximeter.

In the happy case, oximeter inserts the data, and then comes back for more, until all the available data is consumed. The producer uses the next page token (or a sequence number in the request) as an ACK from oximeter that it has collected and handled the last page of data.

In the sad case, oximeter will start making requests to the producer that either lack a page token, or that have the same page token repeated. The producer will again serve the last batch of data, which it has kept on hand.

Batching

There's a wrinkle here, which is that oximeter currently batches up data for inserting into ClickHouse. This is just for efficiency, since the documentation is pretty clear (and we have confirmation from friends of Oxide), that data is best updated in large batches. ("Large" varies, but is on the order of thousands, with the goal being only a handful of inserts per second.)

oximeter would need (I think) to change this strategy. In particular, we can't consider data acknowledged when it's in the batch queue, only when it's in the database. That means the producer would need to keep as much data as fits in a batch, not just the last page. I don't think that's a huge deal, since batches are currently inserted every 5 seconds or 1000 records, whichever comes first. But it does mean we might need a way for the producer to hold on to multiple pages, or a way for oximeter to acknowledge multiple pages.

This sort of suggests a "two-pointer" kind of approach. We use the page token to keep track of the front-running pointer, keeping track of the last data oximeter collected from the producer. These all include sequence numbers in them. oximeter then separately acks the set of sequence numbers, or the last one, after inserting it into the database. The producer then drops that data.

Note that unordered IDs might be better than sequence numbers. If the producer dies, I don't believe we want that starting over from zero, since oximeter is somewhere much further along. Although it's feasible oximeter could just take as fiat whatever sequence numbers it gets. I'm not sure.