linkerd / linkerd/linkerd2

Missing client_id label on inbound requests to destination controller port 8090

Open
#7,861 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

pinned priority/triage
Dominant language
Go
Stars
11.5k
Forks
1.4k
Avg merge
3d 22h
Merged PRs (30d)
53

Description

### What is the issue?

I'm not seeing a `client_id` label on any of the `response_total` stats that are exported by the inbound proxy of the `linkerd-destination` pod when the target port is 8090 (policy), but I am seeing that label set when the target port is 8086 (destination).

It's a bit easier to illustrate with a comparison of these two promql queries:

image

Maybe this is intentional? Without a `client_id` label set for requests to 8090, however, we can't dedupe traffic to that port.

### How can it be reproduced?

Install linkerd and linkerd-viz, then:

```
kubectl -n linkerd-viz port-forward svc/prometheus 9090
```

Visit http://localhost:9090 and run the following query:

```
sum(response_total{direction="inbound", deployment="linkerd-destination"}) by (target_port, client_id)
```

### Logs, error output, etc

See above

### output of `linkerd check -o short`

```
Linkerd core checks
===================

linkerd-identity
----------------
‼ issuer cert is valid for at least 60 days
issuer certificate will expire on 2022-03-10T08:39:24Z
see https://linkerd.io/2/checks/#l5d-identity-issuer-cert-not-expiring-soon for hints

Linkerd extensions checks
=========================

linkerd-multicluster
--------------------
× service mirror controller has required permissions
missing ServiceAccounts: linkerd-service-mirror-aks-viz
missing ClusterRoles: linkerd-service-mirror-access-local-resources-aks-viz
missing ClusterRoleBindings: linkerd-service-mirror-access-local-resources-aks-viz
missing Roles: linkerd-service-mirror-read-remote-creds-aks-viz
missing RoleBindings: linkerd-service-mirror-read-remote-creds-aks-viz
see https://linkerd.io/2/checks/#l5d-multicluster-source-rbac-correct for hints
× service mirror controllers are running
* no service mirror controller deployment for Link aks-viz
see https://linkerd.io/2/checks/#l5d-multicluster-service-mirror-running for hints

Status check results are ×
```

### Environment

```
$ linkerd version --short
edge-22.2.1
edge-22.2.1
```

```
$ kubectl version --short
Client Version: v1.22.5
Server Version: v1.21.2
```

### Possible solution

This might be working as expected, in which case we can close it.

### Additional context

_No response_

### Would you like to work on fixing this bug?

no

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the issue by installing Linkerd and linkerd-viz, port-forwarding Prometheus on port 9090, and running the provided response_total query. Compare inbound metrics for target ports 8090 and 8086; done means requests to port 8090 expose the expected client_id label, or the behavior is documented as intentional.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, prometheus
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.