Idea: max rate of out-of-order samples in a single series

Proposal

The out-of-order feature in Prometheus is aimed at the case where most samples are in-order but a few arrive out-of-order. However I see about one case a week where someone has contrived to send thousands of OOO samples and hence TSDB has lots of small chunks to manage, pushing up CPU and memory usage.

I believe these are mostly the result of misconfiguration, such as 'labeldrop' rules which make separate series collide. Unfortunately, when the data traverses a long pathway to get to TSDB it can be hard to track down the problem.

I propose that there be a maximum rate of OOO samples in any one series, for example 10 in a minute. This can be implemented by storing the time when the last OOO chunk was created in memSeriesOOOFields and checking each time a new chunk is cut (by default an OOO chunk holds 32 samples).

Creating an error will highlight the problem and avoid the additional expense.