Telemetry Service: Metrics

- [ ] We need to send system + application level metrics. Primary data sources -
- [ ] Pilot
- [ ] Central
- [ ] Atlas
- [ ] All other services (Ceph, Boat, Object Storage etc.)
- [ ] Data exporters can be embedded in software, like pilot's own monitoring can send data directly, that's much better than running another daemon. If that's not possible for any specific software, then dameon service can be considered.
- [ ] Ingest endpoint need to be generic : Prometheus doesn't decide any specific schema, only have specific data types - Counter, Gauge, Histogram, and Summary. The exporters can decide the name, params to include. We should keep our ingest endpoint also generic enough, so that for writing the data collector, no changes needed on entral service end.
- [ ] Pilot Specific Extra Telemetry
- [ ] Send the Slow Query / Slow Endpoint metrics to telemetry (consent based). Pilot has already some code for normalizing query / endpoints. (maybe actual data can be shipped by the log service, anyway we need that for drill down)
- [ ] Event Collector like https://github.com/frappe/pulse
- [ ] Easy connectivity with Insights. If we can chose some time series database which supports SQL or if there is adapter for SQL query, then it will be quite easy to build connectivity with insights.

Additional :
- [ ] Alert System : It need to be very easy to configure alert based on rules like prometheus's alertmanager.

### Implementations Findings.

#### Try out
There are other metric storage and service providers that we need to check once.
- [Victoria Metrics](https://victoriametrics.com) which is actively being used by [others](https://zerodha.tech/blog/infra-monitoring-at-zerodha/) as well.
- Implementing Victoria Metrics would require a SQL to PromQL converter so that this works with insights.
- Currently trying our ClickHouse and DuckDB, however both come with their problems.