microsoft / microsoft/retina

networkobservability_tcp_connection_remote only emits address=AllIPs instead of per-remote address labels

Open
#2,559 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
3.2k
Forks
304
Avg merge
1d 19h
Merged PRs (30d)
78

Description

## Summary

`networkobservability_tcp_connection_remote` is documented as a per-remote IP/port metric, but the current implementation only emits a single aggregated series with `address="AllIPs"`.

We are specifically interested in having this supported in the lighter Hubble-based Retina deployment as well, since that is the mode we are using in production.

## Expected behavior

The metric description and docs suggest this should expose one series per active remote address/port pair, for example:

- `networkobservability_tcp_connection_remote{address="10.0.0.1",port="443"}`
- `networkobservability_tcp_connection_remote{address="10.0.0.2",port="5432"}`

From our perspective, this would be especially valuable if available in the Hubble-based Retina deployment, not just in a heavier/full Retina setup.

## Actual behavior

The current code sums all valid remote addresses into a single total and exports only:

- `networkobservability_tcp_connection_remote{address="AllIPs"}`

Relevant code:

- https://github.com/microsoft/retina/blob/577d9179b53c61697b5caf309fb4508244083146/pkg/plugin/linuxutil/netstat_stats_linux.go#L236-L245

The constant is also defined here:

- https://github.com/microsoft/retina/blob/577d9179b53c61697b5caf309fb4508244083146/pkg/plugin/linuxutil/netstat_stats_linux.go#L19-L23

## Why this looks incomplete

The implementation already builds `socketByRemoteAddr` in `processSocks()`, but `updateMetrics()` only validates and sums those entries instead of exporting them individually.

It seems likely the missing piece is handling series cleanup across polls, i.e. setting previously seen remote address/port label combinations to `0` or otherwise removing stale series when they disappear from the current snapshot.

## Impact

This makes the metric much less useful for diagnosing network issues. Dashboards that group `by(address)` end up with a single `AllIPs` series instead of a breakdown by remote endpoint.

In our case, this is affecting a Hubble-based Retina deployment where we want a lightweight but still actionable TCP remote-endpoint diagnostic signal.

## Docs mismatch

These docs describe the metric as per remote IP/port:

- https://github.com/microsoft/retina/blob/main/docs/03-Metrics/02-hubble_metrics.md
- https://github.com/microsoft/retina/blob/main/docs/03-Metrics/modes/basic.md

## Suggested fix

Either:

1. Implement per-remote address/port series export for `networkobservability_tcp_connection_remote`, including stale-series handling between polls

or

2. Update the docs/metric description to reflect that the metric is intentionally aggregated into `AllIPs`

A real per-remote series would be much more useful if cardinality is acceptable or can be gated/configured, and we would especially like to see that supported in the Hubble-based Retina deployment path.

Contributor guide

Open the contributing guide

Research direction

Start in pkg/plugin/linuxutil/netstat_stats_linux.go, reading processSocks(), updateMetrics(), and the AllIPs constant at the referenced lines. Compare the metric expectations in docs/03-Metrics/02-hubble_metrics.md and docs/03-Metrics/modes/basic.md, then determine the intended Hubble deployment behavior. Done means either per-remote address/port series with stale-series handling or documentation that accurately describes aggregation.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
networking, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.