duplicate Hubble DNS flows and ingress/egress labeling bug when DNS request lands on another Node

For a Retina cluster with Hubble enabled, when a Pod makes a DNS request (to the kube-dns service), if the request lands on a coredns Pod on another Node, then running hubble observe flows will produce double the necessary number of flows. Additionally, there seems to be a small bug in labeling the traffic as ingress versus egress.

In the DNS experiment below, ***there exist identical copies of DNS flows (looking at JSON format)***, resulting in 48 DNS flows between `toolbox-pod` and `coredns-5fc6484dd7-bnzlf`, while there are 24 UDP packets when adding packets leaving Node 1 (src=toolbox, dst=coredns), entering Node 2 (src=toolbox, dst=coredns), leaving Node 2 (dst=toolbox, src=coredns), then entering Node 1 (dst=toolbox, src=coredns).

Also in this experiment, ***there seems to be a bug as exemplified below***. A single packet travels from `toolbox` to `coredns-5fc6484dd7-bnzlf`, and a single packet is sent back. Yet there is this ***discrepancy in egress vs. ingress counts***. Presumably, the first and second flows are traced on one Node, and the third and fourth flows are traced on the other Node.

_toolbox -> bnzlf_
1. EGRESS, TO_NETWORK
2. EGRESS, TO_NETWORK
3. EGRESS, TO_NETWORK
4. INGRESS, FROM_NETWORK

_toolbox <- bnzlf_
1. EGRESS, TO_NETWORK
2. EGRESS, TO_NETWORK
3. INGRESS, FROM_NETWORK
4. INGRESS, FROM_NETWORK

### Setup

- Two-node AKS cluster.
- Agent image: `mcr.microsoft.com/containernetworking/retina-agent-enterprise:v0.0.8`
- Relay image: `mcr.microsoft.com/oss/cilium/hubble-relay:v1.15.0`
- Below `toolbox-pod`:
```yaml
apiVersion: v1
kind: Pod
metadata:
name: toolbox-pod
spec:
containers:
- name: toolbox-container
image: acnpublic.azurecr.io/toolbox
ports:
- containerPort: 80
```

### DNS Experiment

#### Pods, Nodes, and IPs

We will focus on `coredns-5fc6484dd7-bnzlf`, which is on a different Node than `toolbox-pod`.

```shell
$ k get po -owide
NAME READY STATUS RESTARTS AGE IP NODE
toolbox-pod 1/1 Running 0 24h 10.224.0.106 aks-nodepool1-32351195-vmss000000
```

```shell
$ k get po -n kube-system -owide
NAME READY STATUS RESTARTS AGE IP NODE
coredns-5fc6484dd7-bnzlf 1/1 Running 0 27h 10.224.1.85 aks-nodepool1-32351195-vmss000001
coredns-5fc6484dd7-k8vl4 1/1 Running 0 27h 10.224.0.189 aks-nodepool1-32351195-vmss000000
```

#### Experiment

Apply these iptables rules to each node to count UDP packets between the two Pods:
```shell
iptables -I FORWARD -d 10.224.0.106 -s 10.224.1.85 --protocol udp --sport 53
iptables -I FORWARD -s 10.224.0.106 -d 10.224.1.85 --protocol udp --dport 53
```

Curl bing.com
```shell
$ k exec toolbox-pod -- curl bing.com
```

Before curling, start observing flows (focus only on the flows to/from `coredns-5fc6484dd7-bnzlf`):
```shell
$ hubble observe flows -f --protocol dns | grep bing.com | grep -v "kube-system/kube-dns"
Aug 6 23:50:54.601: toolbox-pod (ID:1679) -> coredns-5fc6484dd7-bnzlf (ID:4330) dns-request FORWARDED (DNS Query bing.com.default.svc.cluster.local. A)
Aug 6 23:50:54.601: toolbox-pod (ID:1679) -> coredns-5fc6484dd7-bnzlf (ID:4330) dns-request FORWARDED (DNS Query bing.com.default.svc.cluster.local. A)
Aug 6 23:50:54.601: toolbox-pod (ID:1679) -> coredns-5fc6484dd7-bnzlf (ID:4330) dns-request FORWARDED (DNS Query bing.com.default.svc.cluster.local. AAAA)
Aug 6 23:50:54.601: toolbox-pod (ID:1679) -> coredns-5fc6484dd7-bnzlf (ID:4330) dns-request FORWARDED (DNS Query bing.com.default.svc.cluster.local. AAAA)
Aug 6 23:50:54.604: toolbox-pod (ID:1679) -> coredns-5fc6484dd7-bnzlf (ID:4330) dns-request FORWARDED (DNS Query bing.com.default.svc.cluster.local. A)
Aug 6 23:50:54.604: toolbox-pod (ID:1679) -> coredns-5fc6484dd7-bnzlf (ID:4330) dns-request FORWARDED (DNS Query bing.com.default.svc.cluster.local. AAAA)
Aug 6 23:50:54.604: toolbox-pod (ID:1679) -> coredns-5fc6484dd7-bnzlf (ID:4330) dns-request FORWARDED (DNS Query bing.com.default.svc.cluster.local. A)
Aug 6 23:50:54.604: toolbox-pod (ID:1679) -> coredns-5fc6484dd7-bnzlf (ID:4330) dns-request FORWARDED (DNS Query bing.com.default.svc.cluster.local. AAAA)
Aug 6 23:50:54.605: toolbox-pod (ID:1679) <- coredns-5fc6484dd7-bnzlf (ID:4330) dns-response FORWARDED (DNS Answer RCode: Non-Existent Domain (Query bing.com.default.svc.cluster.local. AAAA))
Aug 6 23:50:54.605: toolbox-pod (ID:1679) <- coredns-5fc6484dd7-bnzlf (ID:4330) dns-response FORWARDED (DNS Answer RCode: Non-Existent Domain (Query bing.com.default.svc.cluster.local. AAAA))
Aug 6 23:50:54.605: toolbox-pod (ID:1679) <- coredns-5fc6484dd7-bnzlf (ID:4330) dns-response FORWARDED (DNS Answer RCode: Non-Existent Domain (Query bing.com.default.svc.cluster.local. AAAA))
Aug 6 23:50:54.605: toolbox-pod (ID:1679) <- coredns-5fc6484dd7-bnzlf (ID:4330) dns-response FORWARDED (DNS Answer RCode: Non-Existent Domain (Query bing.com.default.svc.cluster.local. A))
Aug 6 23:50:54.605: toolbox-pod (ID:1679) <- coredns-5fc6484dd7-bnzlf (ID:4330) dns-response FORWARDED (DNS Answer RCode: Non-Existent Domain (Query bing.com.default.svc.cluster.local. A))
Aug 6 23:50:54.605: toolbox-pod (ID:1679) <- coredns-5fc6484dd7-bnzlf (ID:4330) dns-response FORWARDED (DNS Answer RCode: Non-Existent Domain (Query bing.com.default.svc.cluster.local. A))
Aug 6 23:50:54.613: toolbox-pod (ID:1679) <- coredns-5fc6484dd7-bnzlf (ID:4330) dns-response FORWARDED (DNS Answer RCode: Non-Existent Domain (Query bing.com.default.svc.cluster.local. AAAA))
Aug 6 23:50:54.613: toolbox-pod (ID:1679) <- coredns-5fc6484dd7-bnzlf (ID:4330) dns-response FORWARDED (DNS Answer RCode: Non-Existent Domain (Query bing.com.default.svc.cluster.local. A))
Aug 6 23:50:54.614: toolbox-pod (ID:1679) -> coredns-5fc6484dd7-bnzlf (ID:4330) dns-request FORWARDED (DNS Query bing.com.svc.cluster.local. A)
Aug 6 23:50:54.614: toolbox-pod (ID:1679) -> coredns-5fc6484dd7-bnzlf (ID:4330) dns-request FORWARDED (DNS Query bing.com.svc.cluster.local. A)
Aug 6 23:50:54.614: toolbox-pod (ID:1679) -> coredns-5fc6484dd7-bnzlf (ID:4330) dns-request FORWARDED (DNS Query bing.com.svc.cluster.local. AAAA)
Aug 6 23:50:54.614: toolbox-pod (ID:1679) -> coredns-5fc6484dd7-bnzlf (ID:4330) dns-request FORWARDED (DNS Query bing.com.svc.cluster.local. AAAA)
Aug 6 23:50:54.616: toolbox-pod (ID:1679) <- coredns-5fc6484dd7-bnzlf (ID:4330) dns-response FORWARDED (DNS Answer RCode: Non-Existent Domain (Query bing.com.svc.cluster.local. AAAA))
Aug 6 23:50:54.616: toolbox-pod (ID:1679) <- coredns-5fc6484dd7-bnzlf (ID:4330) dns-response FORWARDED (DNS Answer RCode: Non-Existent Domain (Query bing.com.svc.cluster.local. A))
Aug 6 23:50:54.617: toolbox-pod (ID:1679) -> coredns-5fc6484dd7-bnzlf (ID:4330) dns-request FORWARDED (DNS Query bing.com.svc.cluster.local. A)
Aug 6 23:50:54.617: toolbox-pod (ID:1679) -> coredns-5fc6484dd7-bnzlf (ID:4330) dns-request FORWARDED (DNS Query bing.com.svc.cluster.local. AAAA)
Aug 6 23:50:54.617: toolbox-pod (ID:1679) -> coredns-5fc6484dd7-bnzlf (ID:4330) dns-request FORWARDED (DNS Query bing.com.cluster.local. A)
Aug 6 23:50:54.617: toolbox-pod (ID:1679) -> coredns-5fc6484dd7-bnzlf (ID:4330) dns-request FORWARDED (DNS Query bing.com.cluster.local. A)
Aug 6 23:50:54.617: toolbox-pod (ID:1679) -> coredns-5fc6484dd7-bnzlf (ID:4330) dns-request FORWARDED (DNS Query bing.com.svc.cluster.local. A)
Aug 6 23:50:54.617: toolbox-pod (ID:1679) -> coredns-5fc6484dd7-bnzlf (ID:4330) dns-request FORWARDED (DNS Query bing.com.cluster.local. AAAA)
Aug 6 23:50:54.617: toolbox-pod (ID:1679) -> coredns-5fc6484dd7-bnzlf (ID:4330) dns-request FORWARDED (DNS Query bing.com.svc.cluster.local. AAAA)
Aug 6 23:50:54.617: toolbox-pod (ID:1679) -> coredns-5fc6484dd7-bnzlf (ID:4330) dns-request FORWARDED (DNS Query bing.com.cluster.local. AAAA)
Aug 6 23:50:54.618: toolbox-pod (ID:1679) <- coredns-5fc6484dd7-bnzlf (ID:4330) dns-response FORWARDED (DNS Answer RCode: Non-Existent Domain (Query bing.com.svc.cluster.local. AAAA))
Aug 6 23:50:54.618: toolbox-pod (ID:1679) <- coredns-5fc6484dd7-bnzlf (ID:4330) dns-response FORWARDED (DNS Answer RCode: Non-Existent Domain (Query bing.com.svc.cluster.local. AAAA))
Aug 6 23:50:54.618: toolbox-pod (ID:1679) <- coredns-5fc6484dd7-bnzlf (ID:4330) dns-response FORWARDED (DNS Answer RCode: Non-Existent Domain (Query bing.com.svc.cluster.local. AAAA))
Aug 6 23:50:54.618: toolbox-pod (ID:1679) <- coredns-5fc6484dd7-bnzlf (ID:4330) dns-response FORWARDED (DNS Answer RCode: Non-Existent Domain (Query bing.com.svc.cluster.local. A))
Aug 6 23:50:54.618: toolbox-pod (ID:1679) <- coredns-5fc6484dd7-bnzlf (ID:4330) dns-response FORWARDED (DNS Answer RCode: Non-Existent Domain (Query bing.com.svc.cluster.local. A))
Aug 6 23:50:54.618: toolbox-pod (ID:1679) <- coredns-5fc6484dd7-bnzlf (ID:4330) dns-response FORWARDED (DNS Answer RCode: Non-Existent Domain (Query bing.com.svc.cluster.local. A))
Aug 6 23:50:54.619: toolbox-pod (ID:1679) <- coredns-5fc6484dd7-bnzlf (ID:4330) dns-response FORWARDED (DNS Answer RCode: Non-Existent Domain (Query bing.com.cluster.local. A))
Aug 6 23:50:54.619: toolbox-pod (ID:1679) <- coredns-5fc6484dd7-bnzlf (ID:4330) dns-response FORWARDED (DNS Answer RCode: Non-Existent Domain (Query bing.com.cluster.local. AAAA))
Aug 6 23:50:54.619: toolbox-pod (ID:1679) -> coredns-5fc6484dd7-k8vl4 (ID:4330) dns-request FORWARDED (DNS Query bing.com.lxgomovm4wuubbz0evyk1mkg3b.xx.internal.cloudapp.net. A)
Aug 6 23:50:54.619: toolbox-pod (ID:1679) -> coredns-5fc6484dd7-k8vl4 (ID:4330) dns-request FORWARDED (DNS Query bing.com.lxgomovm4wuubbz0evyk1mkg3b.xx.internal.cloudapp.net. AAAA)
Aug 6 23:50:54.619: coredns-5fc6484dd7-k8vl4 (ID:4330) -> 168.63.129.16 (world) dns-request FORWARDED (DNS Query bing.com.lxgomovm4wuubbz0evyk1mkg3b.xx.internal.cloudapp.net. A)
Aug 6 23:50:54.619: 10.224.0.4 (host) -> 168.63.129.16 (world) dns-request FORWARDED (DNS Query bing.com.lxgomovm4wuubbz0evyk1mkg3b.xx.internal.cloudapp.net. A)
Aug 6 23:50:54.619: 10.224.0.4 (host) -> 168.63.129.16 (world) dns-request FORWARDED (DNS Query bing.com.lxgomovm4wuubbz0evyk1mkg3b.xx.internal.cloudapp.net. A)
Aug 6 23:50:54.620: coredns-5fc6484dd7-k8vl4 (ID:4330) -> 168.63.129.16 (world) dns-request FORWARDED (DNS Query bing.com.lxgomovm4wuubbz0evyk1mkg3b.xx.internal.cloudapp.net. AAAA)
Aug 6 23:50:54.620: 10.224.0.4 (host) -> 168.63.129.16 (world) dns-request FORWARDED (DNS Query bing.com.lxgomovm4wuubbz0evyk1mkg3b.xx.internal.cloudapp.net. AAAA)
Aug 6 23:50:54.620: 10.224.0.4 (host) -> 168.63.129.16 (world) dns-request FORWARDED (DNS Query bing.com.lxgomovm4wuubbz0evyk1mkg3b.xx.internal.cloudapp.net. AAAA)
Aug 6 23:50:54.620: toolbox-pod (ID:1679) -> coredns-5fc6484dd7-bnzlf (ID:4330) dns-request FORWARDED (DNS Query bing.com.cluster.local. A)
Aug 6 23:50:54.620: toolbox-pod (ID:1679) -> coredns-5fc6484dd7-bnzlf (ID:4330) dns-request FORWARDED (DNS Query bing.com.cluster.local. AAAA)
Aug 6 23:50:54.620: toolbox-pod (ID:1679) -> coredns-5fc6484dd7-bnzlf (ID:4330) dns-request FORWARDED (DNS Query bing.com.cluster.local. A)
Aug 6 23:50:54.620: toolbox-pod (ID:1679) -> coredns-5fc6484dd7-bnzlf (ID:4330) dns-request FORWARDED (DNS Query bing.com.cluster.local. AAAA)
Aug 6 23:50:54.621: toolbox-pod (ID:1679) <- coredns-5fc6484dd7-bnzlf (ID:4330) dns-response FORWARDED (DNS Answer RCode: Non-Existent Domain (Query bing.com.cluster.local. A))
Aug 6 23:50:54.621: toolbox-pod (ID:1679) <- coredns-5fc6484dd7-bnzlf (ID:4330) dns-response FORWARDED (DNS Answer RCode: Non-Existent Domain (Query bing.com.cluster.local. A))
Aug 6 23:50:54.621: toolbox-pod (ID:1679) <- coredns-5fc6484dd7-bnzlf (ID:4330) dns-response FORWARDED (DNS Answer RCode: Non-Existent Domain (Query bing.com.cluster.local. A))
Aug 6 23:50:54.621: toolbox-pod (ID:1679) <- coredns-5fc6484dd7-bnzlf (ID:4330) dns-response FORWARDED (DNS Answer RCode: Non-Existent Domain (Query bing.com.cluster.local. AAAA))
Aug 6 23:50:54.621: toolbox-pod (ID:1679) <- coredns-5fc6484dd7-bnzlf (ID:4330) dns-response FORWARDED (DNS Answer RCode: Non-Existent Domain (Query bing.com.cluster.local. AAAA))
Aug 6 23:50:54.621: toolbox-pod (ID:1679) <- coredns-5fc6484dd7-bnzlf (ID:4330) dns-response FORWARDED (DNS Answer RCode: Non-Existent Domain (Query bing.com.cluster.local. AAAA))
Aug 6 23:50:54.622: 10.224.0.4 (host) <- 168.63.129.16 (world) dns-response FORWARDED (DNS Answer RCode: Non-Existent Domain (Query bing.com.lxgomovm4wuubbz0evyk1mkg3b.xx.internal.cloudapp.net. A))
Aug 6 23:50:54.622: coredns-5fc6484dd7-k8vl4 (ID:4330) <- 168.63.129.16 (world) dns-response FORWARDED (DNS Answer RCode: Non-Existent Domain (Query bing.com.lxgomovm4wuubbz0evyk1mkg3b.xx.internal.cloudapp.net. A))
```

The packets sent/received were:

**Node 0**:
```shell
root@aks-nodepool1-32351195-vmss000000:/# iptables -vnL FORWARD | grep 10.224.0.106
# Warning: iptables-legacy tables present, use iptables-legacy to see them
0 0 udp -- * * 10.224.1.85 10.224.0.106 udp spt:53
0 0 udp -- * * 10.224.0.106 10.224.1.85 udp dpt:53
root@aks-nodepool1-32351195-vmss000000:/#
root@aks-nodepool1-32351195-vmss000000:/# iptables -vnL FORWARD | grep 10.224.0.106
# Warning: iptables-legacy tables present, use iptables-legacy to see them
6 998 udp -- * * 10.224.1.85 10.224.0.106 udp spt:53
6 440 udp -- * * 10.224.0.106 10.224.1.85 udp dpt:53
```

**Node 1**:
```shell
root@aks-nodepool1-32351195-vmss000001:/# iptables -vnL FORWARD | grep 10.224.0.106
# Warning: iptables-legacy tables present, use iptables-legacy to see them
0 0 udp -- * * 10.224.0.106 10.224.1.85 udp dpt:53
0 0 udp -- * * 10.224.1.85 10.224.0.106 udp spt:53
root@aks-nodepool1-32351195-vmss000001:/#
root@aks-nodepool1-32351195-vmss000001:/# iptables -vnL FORWARD | grep 10.224.0.106
# Warning: iptables-legacy tables present, use iptables-legacy to see them
6 440 udp -- * * 10.224.0.106 10.224.1.85 udp dpt:53
6 998 udp -- * * 10.224.1.85 10.224.0.106 udp spt:53
```

When inspecting JSON output of the flows (`hubble observe -ojson`), we see the flows are duplicated and there exists the ingress/egress labeling bug described at the top. Here is an example looking at the `tester.default.svc.cluster.local` query:
```json
====TOOLBOX -> BNZLF====
{"flow":{"time":"2024-08-06T20:09:10.316833065Z","verdict":"FORWARDED","IP":{"source":"10.224.0.106","destination":"10.224.1.85","ipVersion":"IPv4"},"source":{"ID":1679,"identity":1679,"namespace":"default","labels":[],"pod_name":"toolbox-pod"},"destination":{"ID":4330,"identity":4330,"namespace":"kube-system","labels":[],"pod_name":"coredns-5fc6484dd7-bnzlf"},"Type":"L7","l7":{"type":"REQUEST","dns":{"query":"tester.default.svc.cluster.local.","rcode":24,"qtypes":["A"]}},"event_type":{"type":129},"source_service":{},"destination_service":{},"traffic_direction":"EGRESS","trace_observation_point":"TO_NETWORK","is_reply":false,"Summary":"DNS Query tester.default.svc.cluster.local. A"},"time":"2024-08-06T20:09:10.316833065Z"}
{"flow":{"time":"2024-08-06T20:09:10.316848165Z","verdict":"FORWARDED","IP":{"source":"10.224.0.106","destination":"10.224.1.85","ipVersion":"IPv4"},"source":{"ID":1679,"identity":1679,"namespace":"default","labels":[],"pod_name":"toolbox-pod"},"destination":{"ID":4330,"identity":4330,"namespace":"kube-system","labels":[],"pod_name":"coredns-5fc6484dd7-bnzlf"},"Type":"L7","l7":{"type":"REQUEST","dns":{"query":"tester.default.svc.cluster.local.","rcode":24,"qtypes":["A"]}},"event_type":{"type":129},"source_service":{},"destination_service":{},"traffic_direction":"EGRESS","trace_observation_point":"TO_NETWORK","is_reply":false,"Summary":"DNS Query tester.default.svc.cluster.local. A"},"time":"2024-08-06T20:09:10.316848165Z"}
{"flow":{"time":"2024-08-06T20:09:10.320490428Z","verdict":"FORWARDED","IP":{"source":"10.224.0.106","destination":"10.224.1.85","ipVersion":"IPv4"},"source":{"ID":1679,"identity":1679,"namespace":"default","labels":[],"pod_name":"toolbox-pod"},"destination":{"ID":4330,"identity":4330,"namespace":"kube-system","labels":[],"pod_name":"coredns-5fc6484dd7-bnzlf"},"Type":"L7","l7":{"type":"REQUEST","dns":{"query":"tester.default.svc.cluster.local.","rcode":24,"qtypes":["A"]}},"event_type":{"type":129},"source_service":{},"destination_service":{},"traffic_direction":"EGRESS","trace_observation_point":"TO_NETWORK","is_reply":false,"Summary":"DNS Query tester.default.svc.cluster.local. A"},"time":"2024-08-06T20:09:10.320490428Z"}
{"flow":{"time":"2024-08-06T20:09:10.320385227Z","verdict":"FORWARDED","IP":{"source":"10.224.0.106","destination":"10.224.1.85","ipVersion":"IPv4"},"source":{"ID":1679,"identity":1679,"namespace":"default","labels":[],"pod_name":"toolbox-pod"},"destination":{"ID":4330,"identity":4330,"namespace":"kube-system","labels":[],"pod_name":"coredns-5fc6484dd7-bnzlf"},"Type":"L7","l7":{"type":"REQUEST","dns":{"query":"tester.default.svc.cluster.local.","rcode":24,"qtypes":["A"]}},"event_type":{"type":129},"source_service":{},"destination_service":{},"traffic_direction":"INGRESS","trace_observation_point":"FROM_NETWORK","is_reply":false,"Summary":"DNS Query tester.default.svc.cluster.local. A"},"time":"2024-08-06T20:09:10.320385227Z"}

====TOOLBOX <- BNZLF====
{"flow":{"time":"2024-08-06T20:09:10.320865029Z","verdict":"FORWARDED","IP":{"source":"10.224.1.85","destination":"10.224.0.106","ipVersion":"IPv4"},"source":{"ID":4330,"identity":4330,"namespace":"kube-system","labels":[],"pod_name":"coredns-5fc6484dd7-bnzlf"},"destination":{"ID":1679,"identity":1679,"namespace":"default","labels":[],"pod_name":"toolbox-pod"},"Type":"L7","l7":{"type":"RESPONSE","dns":{"query":"tester.default.svc.cluster.local.","rcode":3,"qtypes":["A"]}},"event_type":{"type":129},"source_service":{},"destination_service":{},"traffic_direction":"EGRESS","trace_observation_point":"TO_NETWORK","is_reply":true,"Summary":"DNS Answer RCode: Non-Existent Domain (Query tester.default.svc.cluster.local. A)"},"time":"2024-08-06T20:09:10.320865029Z"}
{"flow":{"time":"2024-08-06T20:09:10.320871330Z","verdict":"FORWARDED","IP":{"source":"10.224.1.85","destination":"10.224.0.106","ipVersion":"IPv4"},"source":{"ID":4330,"identity":4330,"namespace":"kube-system","labels":[],"pod_name":"coredns-5fc6484dd7-bnzlf"},"destination":{"ID":1679,"identity":1679,"namespace":"default","labels":[],"pod_name":"toolbox-pod"},"Type":"L7","l7":{"type":"RESPONSE","dns":{"query":"tester.default.svc.cluster.local.","rcode":3,"qtypes":["A"]}},"event_type":{"type":129},"source_service":{},"destination_service":{},"traffic_direction":"EGRESS","trace_observation_point":"TO_NETWORK","is_reply":true,"Summary":"DNS Answer RCode: Non-Existent Domain (Query tester.default.svc.cluster.local. A)"},"time":"2024-08-06T20:09:10.320871330Z"}
{"flow":{"time":"2024-08-06T20:09:10.318756683Z","verdict":"FORWARDED","IP":{"source":"10.224.1.85","destination":"10.224.0.106","ipVersion":"IPv4"},"source":{"ID":4330,"identity":4330,"namespace":"kube-system","labels":[],"pod_name":"coredns-5fc6484dd7-bnzlf"},"destination":{"ID":1679,"identity":1679,"namespace":"default","labels":[],"pod_name":"toolbox-pod"},"Type":"L7","l7":{"type":"RESPONSE","dns":{"query":"tester.default.svc.cluster.local.","rcode":3,"qtypes":["A"]}},"event_type":{"type":129},"source_service":{},"destination_service":{},"traffic_direction":"INGRESS","trace_observation_point":"FROM_NETWORK","is_reply":true,"Summary":"DNS Answer RCode: Non-Existent Domain (Querytester.default.svc.cluster.local. A)"},"time":"2024-08-06T20:09:10.318756683Z"}
{"flow":{"time":"2024-08-06T20:09:10.320806929Z","verdict":"FORWARDED","IP":{"source":"10.224.1.85","destination":"10.224.0.106","ipVersion":"IPv4"},"source":{"ID":4330,"identity":4330,"namespace":"kube-system","labels":[],"pod_name":"coredns-5fc6484dd7-bnzlf"},"destination":{"ID":1679,"identity":1679,"namespace":"default","labels":[],"pod_name":"toolbox-pod"},"Type":"L7","l7":{"type":"RESPONSE","dns":{"query":"tester.default.svc.cluster.local.","rcode":3,"qtypes":["A"]}},"event_type":{"type":129},"source_service":{},"destination_service":{},"traffic_direction":"INGRESS","trace_observation_point":"FROM_NETWORK","is_reply":true,"Summary":"DNS Answer RCode: Non-Existent Domain (Querytester.default.svc.cluster.local. A)"},"time":"2024-08-06T20:09:10.320806929Z"}
```