Prometheus Metrics for TesseraCT

The underlying Tessera and this TesseraCT project is using OpenTelemetry for logging and tracing, which is tied to GCP as a runtime environment. For self-hosted environments, and even for GCP/AWS hosted environments, having telemetry for observability is vital. To run TesseraCT (I am interested in operating a POSIX ctlog), having a /metrics endpoint to scrape with Prometheus is a requirement.

Our colleagues who develop Sunlight have done this very well. A sampling of their telemetry which I think would be a useful starting point for TesseraCT:

addchain_requests_total
addchain_wait_seconds
addchain_wait_seconds_count
addchain_wait_seconds_sum
cache_get_duration_seconds
cache_get_duration_seconds_count
cache_get_duration_seconds_sum
cache_put_duration_seconds
cache_put_duration_seconds_count
cache_put_duration_seconds_sum
cache_put_errors_total
config_notafter_end_timestamp_seconds
config_notafter_start_timestamp_seconds
config_roots_total
fs_op_duration_seconds
fs_op_duration_seconds_count
fs_op_duration_seconds_sum
http_in_flight_requests
http_request_duration_seconds
http_request_duration_seconds_count
http_request_duration_seconds_sum
http_requests_total
issuers_cache_total
sequencing_data_tiles_bytes
sequencing_data_tiles_bytes_count
sequencing_data_tiles_bytes_sum
sequencing_data_tiles_gzip_bytes
sequencing_data_tiles_gzip_bytes_count
sequencing_data_tiles_gzip_bytes_sum
sequencing_duration_seconds
sequencing_duration_seconds_count
sequencing_duration_seconds_sum
sequencing_leaf_bytes
sequencing_leaf_bytes_count
sequencing_leaf_bytes_sum
sequencing_pool_entries
sequencing_pool_entries_count
sequencing_pool_entries_sum
sequencing_rounds_total
sequencing_uploaded_tiles_total
tree_size_leaves_total
tree_timestamp_seconds

A few other things that could be useful:

• Badger DB telemetry.
• Knowing about duplicates will help.
• A notion of checkpoint target (1500ms) and actual (could be larger, signalling contention).

No files or tests are named. Start by locating TesseraCT's existing OpenTelemetry setup and the HTTP entry point for a /metrics endpoint; done means Prometheus can scrape useful request, sequencing, storage, cache, and checkpoint telemetry, including the listed areas.