microsoft / microsoft/retina

Chat-based Network Troubleshooting (data insights for Retina/Hubble)

Open
#439 0 comments 3 reactions 1 assignee Claimed by @huntergregory View on GitHub
area/data-ingestion-and-visualization type/enhancement
Dominant language
Go
Stars
3.2k
Forks
304
Avg merge
1d 16h
Merged PRs (30d)
79

Description

Goal: user can investigate Kubernetes networking issues via a chat tool. The tool determines what data to get, then obtains and analyzes these data.

See example chats below.

### Context and Benefits

Retina provides a few data sources which help to monitor and detect Kubernetes networking issues. Namely, these data sources are:
- Prometheus ***metrics***
- Distributed ***packet captures***
- Traffic ***flow logs*** (per #417)

Having automated insights for these data would:
1. Increase productivity (less time crafting commands/queries and sifting through all the data).
2. Limit the need for in-depth knowledge of networking or the Retina stack.

We can leverage tools such as anomaly detection and LLMs to aid in data analysis. On top of this foundation, we can use LLMs to create a chat-based network troubleshooting experience, which will:
1. Prompt the user for clarifying information.
2. Triage the user's request into a scenario such as:
a. Checking for dropped packets (via metrics).
b. Translating the user request into a packet capture command (`kubectl retina capture`).

### Example Chats

***Disclaimer: this is exploratory work. Actual results may vary.***

#### Dropped Packets

> *USER*: my application is no longer receiving requests
*RETINA*: I can look for dropped packets for your application. Should I look into any namespace, Pod prefix, or Nodes in particular?
*USER*: not sure
*RETINA*: There seems to be an iptables rule dropping incoming traffic for Pod `frontend/web-app`. This seems to be caused by the NetworkPolicy `frontend/block-external-traffic`.

#### DNS Issue

> *USER*: Can't connect to google.com
*RETINA*: I can look at DNS metrics for the query `google.com`. Should I look into any namespace, Pod prefix, or Nodes in particular?
*USER*: backend namespace
*RETINA*: Pod `backend/pod-47` on `node-123` is receiving DNS errors for query `google.com`. Other Pods in the namespace are receiving successful DNS responses for the same query. The DNS issue seems to be specific to `node-123`.

#### Check that Pod A reaches Pod B

> *USER*: check that my web-app deployment in frontend namespace is reaching the database deployment in backend namespace
*RETINA*: Sure, I can check traffic flow logs to see if `frontend/web-app*` Pods are reaching `backend/database*` Pods. How does this sound?
*USER*: yes
*RETINA*: `frontend/web-app*` Pods have sent traffic to `backend/database*` Pods 100 times. However, `backend/database*` Pods have only received traffic from `frontend/web-app*` Pods 5 times. Would you like me to check for dropped packets for `frontend/web-app*` Pods?
*USER*: yes
*RETINA*: Some of the Nodes which have `frontend/web-app*` Pods are dropping packets since the network interface's receive buffer is full. To mitigate, you can reduce network traffic per Node or use Nodes with a higher network bandwidth.

#### Packet Capture Command

> *USER*: can you run a packet capture for the database deployment?
*RETINA*: I can run a packet capture on each Node that has Pods with label `app=database`. The command for this is `kubectl retina capture ...`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.