High "No Response" Metric in Retina Latency Module
- Dominant language
- Go
- Stars
- 3.2k
- Forks
- 304
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 78
Description
### Description
A significant portion of API server requests are being marked as "no response" by Retina's latency metrics, despite correct latency measurement for most traffic. Increasing the TTL did not resolve the issue.
Log level: debug
Following log in retina agent
```
ts=2025-07-23T16:20:56.486Z level=debug caller=metrics/latency.go:128 msg="Evicted item" srcIP=10.224.0.4 dstIP=20.13.226.96 srcPort=49912 dstPort=443 id=1865206169 timestamp=980840618
ts=2025-07-23T16:20:56.486Z level=debug caller=metrics/latency.go:137 msg="Incremented no response metric" metric=adv_node_apiserver_no_response
```
Following metric keeps increasing:
```
# HELP networkobservability_adv_node_apiserver_no_response Number of packets that did not get a response from node apiserver
# TYPE networkobservability_adv_node_apiserver_no_response counter
networkobservability_adv_node_apiserver_no_response{no_response="no_response"} 1095
````
### Potential Causes
- Partial packet trace coverage (response packets not always captured).
- Flow key matching issues (NAT, asymmetric routing, key construction bugs).
- Application-level drops (API server errors, TLS handshake failures).
- Edge case traffic routing not handled in current monitoring setup.
### Steps Taken
- Increased TTL to 2 seconds, observed similar rate of "no response".
- Verified that latency and handshake metrics are correctly populated for majority of traffic.
### Next Steps
- Review and audit flow matching logic in Retina.
- Verify completeness of packet trace/monitoring coverage.
- Correlate "no response" events with API server and kubelet logs.
- Add debug logging in Retina for unmatched flows to gather more context.
### Additional Context
- Monitoring configuration appears to be mostly correct, as latency is measured for ~80% of traffic.
- The remaining "no response" cases may be due to edge case flows or packet capture limitations.
---
**Assignee:**
**Priority:**
**Labels:** investigation, metrics, retina
Contributor guide
Assessment
This issue has not been assessed yet.