networkobservability_tcp_connection_remote only emits address=AllIPs instead of per-remote address labels
- Dominant language
- Go
- Stars
- 3.2k
- Forks
- 304
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 78
Description
## Summary
`networkobservability_tcp_connection_remote` is documented as a per-remote IP/port metric, but the current implementation only emits a single aggregated series with `address="AllIPs"`.
We are specifically interested in having this supported in the lighter Hubble-based Retina deployment as well, since that is the mode we are using in production.
## Expected behavior
The metric description and docs suggest this should expose one series per active remote address/port pair, for example:
- `networkobservability_tcp_connection_remote{address="10.0.0.1",port="443"}`
- `networkobservability_tcp_connection_remote{address="10.0.0.2",port="5432"}`
From our perspective, this would be especially valuable if available in the Hubble-based Retina deployment, not just in a heavier/full Retina setup.
## Actual behavior
The current code sums all valid remote addresses into a single total and exports only:
- `networkobservability_tcp_connection_remote{address="AllIPs"}`
Relevant code:
- https://github.com/microsoft/retina/blob/577d9179b53c61697b5caf309fb4508244083146/pkg/plugin/linuxutil/netstat_stats_linux.go#L236-L245
The constant is also defined here:
- https://github.com/microsoft/retina/blob/577d9179b53c61697b5caf309fb4508244083146/pkg/plugin/linuxutil/netstat_stats_linux.go#L19-L23
## Why this looks incomplete
The implementation already builds `socketByRemoteAddr` in `processSocks()`, but `updateMetrics()` only validates and sums those entries instead of exporting them individually.
It seems likely the missing piece is handling series cleanup across polls, i.e. setting previously seen remote address/port label combinations to `0` or otherwise removing stale series when they disappear from the current snapshot.
## Impact
This makes the metric much less useful for diagnosing network issues. Dashboards that group `by(address)` end up with a single `AllIPs` series instead of a breakdown by remote endpoint.
In our case, this is affecting a Hubble-based Retina deployment where we want a lightweight but still actionable TCP remote-endpoint diagnostic signal.
## Docs mismatch
These docs describe the metric as per remote IP/port:
- https://github.com/microsoft/retina/blob/main/docs/03-Metrics/02-hubble_metrics.md
- https://github.com/microsoft/retina/blob/main/docs/03-Metrics/modes/basic.md
## Suggested fix
Either:
1. Implement per-remote address/port series export for `networkobservability_tcp_connection_remote`, including stale-series handling between polls
or
2. Update the docs/metric description to reflect that the metric is intentionally aggregated into `AllIPs`
A real per-remote series would be much more useful if cardinality is acceptable or can be gated/configured, and we would especially like to see that supported in the Hubble-based Retina deployment path.
Contributor guide
Research direction
Start in pkg/plugin/linuxutil/netstat_stats_linux.go, reading processSocks(), updateMetrics(), and the AllIPs constant at the referenced lines. Compare the metric expectations in docs/03-Metrics/02-hubble_metrics.md and docs/03-Metrics/modes/basic.md, then determine the intended Hubble deployment behavior. Done means either per-remote address/port series with stale-series handling or documentation that accurately describes aggregation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- networking, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100