Evaluate Prometheus resource usage

Background

In #2922 a user reported a linkerd-prometheus pod using 30GB mem and 10GB ephemeral storage. Many factors contribute to Prometheus' resource usage, including:

• total time series (prometheus_tsdb_head_series == ~500k == ~300 linkerd proxies x ~1700 metrics/proxy)
• scrape_interval: 10s
• --storage.tsdb.retention.time=6h
• read load (via linkerd dashboard and Grafana)

Current state

Replicating the above set up with Prometheus v2.10.0 decreased steady state memory usage from 10GB -> 5GB, and high read-load from 12GB -> 8GB, this change will ship in #2979.

Proposal

Evaluate Prometheus resource usage, the goal being one or more of these outcomes:

1. Linkerd default install changes
   • upgrade to Prometheus 2.11 when WAL compression lands
   • decrease set of metrics exported from proxy (or drop during collection)
   • optimize reads from linkerd dashboard and Grafana (via recording rules and/or fewer queries per page)
   • modify storage.tsdb.retention.time
   • modify storage.tsdb.retention.size
   • modify scrape interval
   • ephemeral storage limits
2. Linkerd user tunable settings (via linkerd install)
   • storage.tsdb.retention.time
   • storage.tsdb.retention.size
   • scrape interval
   • ephermal storage limits
3. Document to the user how best to manage resource usage. This could involve modifying the linkerd-prometheus installation to use persistent volumes, etc. (https://github.com/linkerd/linkerd2/issues/2922#issuecomment-504113048)

/cc @jamesallen-vol @suever @complex64 (thanks for the user reports!)

Start by reviewing the Prometheus resource factors and the proposed Linkerd install settings, including retention, scrape interval, and ephemeral storage. Examine the linkerd dashboard and Grafana read load described in the issue. Done means producing a concrete default change, user-tunable setting, or resource-management guidance.