beyond-all-reason / beyond-all-reason/RecoilEngine
Dedicated server network metrics
- Dominant language
- C++
- Stars
- 679
- Forks
- 290
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 38
Description
## Context
### In BAR
We've noticed several occasions of network issues disguised as player lag that have resulted in some debugging rabbitholes when reported (until we eventually go "🤷♂️ guess it was the network").
And in general, if there are presumed network issues (our providers don't seem to be the most reliable) and the best we have to track that is our [smokeping probes](https://github.com/beyond-all-reason/ansible-monitoring/blob/main/roles/client/templates/smokeping_prober.service.j2) on each host, which can tell us of an outright network outage but does little for more transient issues.
If we can add some observability to dedicated that allow us to introspect better into both the symptoms and causes of network issues, then we can make debugging easier and give us more concrete data to inform possible future work.
### In general
While looking into some network issues, I noticed that our bandwidth limit code looks broken: https://discord.com/channels/549281623154229250/1530165600578699427
If we add some observability we can see how much this actually matters in practice :) (but we should fix nonetheless).
## Proposed work
After some discussion with @p2004a [here](https://discord.com/channels/549281623154229250/1529577404928888842) it seems the approach that makes the most sense is to expose a `/metrics` endpoint in the prometheus/openmetrics format, that can then be scraped externally. This also ends up being the simplest to implement within engine, and also becomes easily debuggable locally.
I looked into both opentelemetry and prometheus C++ libraries as options, and prom was the clear winner. Opentelemetry has a ton of abstractions that make it flexible, but also much more complicated to setup in-code. It would make sense if we needed tracing or logging out of otel, but since we only need/want metrics we might as well lean into the (very mature) prometheus library.
Once the prometheus library is setup, adding metrics is fairly straightforward. Observability can get really messy really quickly if a bunch of metric lines are added inline everywhere, so some abstractions should be introduced to keep real code logic isolated from as much observability code as reasonable. In other words, the existing network code shouldn't need to know about prometheus metrics if done well.
## Proposed metrics
1. a basic set of metrics about the currently running game, its participants, game id, etc. so issues in a replay can be correlated to metrics easily.
2. bandwidth/packet counters
3. throttle/stall metrics (since we have code that explicitly throttles players)
4. some sort of metrics for packet loss, retransmissions, etc.
5. RTT/ping time metrics
6. UDP socket failures? (probably less important)
7. internal datastructure info like queue depths? (might help with debugging)
8. a histogram of response times/RTT/ping, in aggregate? (might help us with future geo-locating servers)
9. perhaps reporting on desyncs, dedicated loop processing info, etc? (out of scope of network)
## Cardinality
Napkin math. After implementing some/most of the above, cardinality ends up being in the hundreds of metrics (eg. lets conservatively say 200) per game when per-player/per-connection metrics are off, and then multiplied by num_participants when on.
Right now BAR has in the neighborhood 500 of concurrent games at peak iirc, let's say average players is 4v4=8 players. So 300 * 500 * 8 = 1.2 million active series. Very doable in small victoriametrics instances.
At a future BAR launch date, or if our player count climbs, the number of active games will be much larger. In that case, we have levers for 1. turning off per-player metrics, 2. sizing up victoriametrics, 3. disabling a subset of the metrics (which would need a code change).
## Churn
The big problem we will encounter, speaking from experience, is metric churn. Since games are very short lived, if we give each metric series a unique `instance` label eg. `instance=gameserverabcd123` or `gameid` label and these are not reused, then while our concurrent active series will be low our total series per day will be in the 10s or even 100s of millions. This is similar to if you use k8s pod ids in metrics labels and recycle your pods often (ask me how I know!)
So, we need to be careful and make sure we do two things to keep churn low:
1. any game-unique label should be put into an info series, so the churn is isolated to a single metric for that label. eg. `recoil_server_game_info{gameid="ab38dn3"}`
2. scrape targets must re-use their instance label as much as possible.
- to clarify, the targets must have have a unique label set against *any other concurrently alive scrape target*. So two servers running at the same time must have different instance labels
- but a target with `instance=10.0.0.1:13945` that runs a game, shuts down, can have its `instance=10.0.0.1:13945` reused by a target that lives before or after it.
If we stick to reusing instance labels, then 99% of the metrics won't churn on game finished/server shutdown. The other 1% will be isolated metric series that won't meaningfully raised churn.
Contributor guide
Research direction
The issue names no RecoilEngine files or tests; start by locating the dedicated-server network and bandwidth-limit code. Review the proposed Prometheus/OpenMetrics /metrics endpoint, then define an initial metric scope, abstractions, and label-reuse rules before implementation; done means the agreed metrics are externally scrapeable without excessive series churn.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, networking, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100