Metrics can be lost if backhual is not available for more than 10 minutes [future issue]

Given the current metrics service implementation if we cannot publish metrics over HTTP for more than 10 minutes we will start to lose metrics payloads, this can leave chunks of delta values missing which will cause future metrics to be incorrect. We need to think of a scalable way to preserve metrics if a publish fails, perhaps by cycling metrics back into the timed cache or processing cache if they cannot be immediately sent, an infinitely sized queue could also be used but this will not scale well

Start by reviewing the current metrics service implementation, especially the timed cache and processing cache used around HTTP publishing. Determine how failed publishes are handled and design a scalable preservation approach; done means metrics payloads are not lost during outages longer than 10 minutes and future delta values remain correct.

Written by the indexing model from the issue text.