Missing client_id label on inbound requests to destination controller port 8090
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 11.5k
- Forks
- 1.4k
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 53
Description
### What is the issue?
I'm not seeing a `client_id` label on any of the `response_total` stats that are exported by the inbound proxy of the `linkerd-destination` pod when the target port is 8090 (policy), but I am seeing that label set when the target port is 8086 (destination).
It's a bit easier to illustrate with a comparison of these two promql queries:

Maybe this is intentional? Without a `client_id` label set for requests to 8090, however, we can't dedupe traffic to that port.
### How can it be reproduced?
Install linkerd and linkerd-viz, then:
```
kubectl -n linkerd-viz port-forward svc/prometheus 9090
```
Visit http://localhost:9090 and run the following query:
```
sum(response_total{direction="inbound", deployment="linkerd-destination"}) by (target_port, client_id)
```
### Logs, error output, etc
See above
### output of `linkerd check -o short`
```
Linkerd core checks
===================
linkerd-identity
----------------
‼ issuer cert is valid for at least 60 days
issuer certificate will expire on 2022-03-10T08:39:24Z
see https://linkerd.io/2/checks/#l5d-identity-issuer-cert-not-expiring-soon for hints
Linkerd extensions checks
=========================
linkerd-multicluster
--------------------
× service mirror controller has required permissions
missing ServiceAccounts: linkerd-service-mirror-aks-viz
missing ClusterRoles: linkerd-service-mirror-access-local-resources-aks-viz
missing ClusterRoleBindings: linkerd-service-mirror-access-local-resources-aks-viz
missing Roles: linkerd-service-mirror-read-remote-creds-aks-viz
missing RoleBindings: linkerd-service-mirror-read-remote-creds-aks-viz
see https://linkerd.io/2/checks/#l5d-multicluster-source-rbac-correct for hints
× service mirror controllers are running
* no service mirror controller deployment for Link aks-viz
see https://linkerd.io/2/checks/#l5d-multicluster-service-mirror-running for hints
Status check results are ×
```
### Environment
```
$ linkerd version --short
edge-22.2.1
edge-22.2.1
```
```
$ kubectl version --short
Client Version: v1.22.5
Server Version: v1.21.2
```
### Possible solution
This might be working as expected, in which case we can close it.
### Additional context
_No response_
### Would you like to work on fixing this bug?
no
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the issue by installing Linkerd and linkerd-viz, port-forwarding Prometheus on port 9090, and running the provided response_total query. Compare inbound metrics for target ports 8090 and 8086; done means requests to port 8090 expose the expected client_id label, or the behavior is documented as intentional.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, prometheus
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100