linkerd / linkerd/linkerd2

EOF when running linkerd tap command against a category of resources on a k3d cluster

Open
#5,219 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area/tap bug good first issue help wanted priority/P2
Dominant language
Go
Stars
11.5k
Forks
1.4k
Avg merge
3d 22h
Merged PRs (30d)
53

Description

### What is the issue?
Running the `linkerd tap` command against a control plane deployed to a k3d cluster results in an EOF error when targeting an entire category of kubernetes resources (e.g. Pod or Deployment).

The same command run against a specific resource (deploy/linkerd-prometheus, or a specific pod) works as expected.

### How can it be reproduced?
- Provision a cluster using [k3d](https://k3d.io/) (The number of nodes doesn't matter)
-- `k3d cluster create tap-test`

- Deploy the linkerd control plane
-- `linkerd install | kubectl apply -f -`

- Run tap against all resource types in the linkerd control plane
-- Pods: `linkerd tap -n linkerd po --verbose`
-- Deployments: `linkerd tap -n linkerd deploy --verbose`

### Logs, error output, etc
```
DEBU[0000] Response from [https://0.0.0.0:43867/apis/tap.linkerd.io/v1alpha1/watch/namespaces/linkerd/pods//tap] had headers: map[Cache-Control:[no-cache, private] Content-Length:[205] Content-Type:[application/json] Date:[Thu, 12 Nov 2020 17:58:32 GMT]]
DEBU[0000] Waiting for data...
error reading byte stream header: error while reading bytes from message: unexpected EOF
```

#### `linkerd check` output

```text
kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API

kubernetes-version
------------------
√ is running the minimum Kubernetes API version
√ is running the minimum kubectl version

linkerd-existence
-----------------
√ 'linkerd-config' config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
√ controller pod is running
√ can initialize the client
√ can query the control plane API

linkerd-config
--------------
√ control plane Namespace exists
√ control plane ClusterRoles exist
√ control plane ClusterRoleBindings exist
√ control plane ServiceAccounts exist
√ control plane CustomResourceDefinitions exist
√ control plane MutatingWebhookConfigurations exist
√ control plane ValidatingWebhookConfigurations exist
√ control plane PodSecurityPolicies exist

linkerd-identity
----------------
√ certificate config is valid
√ trust anchors are using supported crypto algorithm
√ trust anchors are within their validity period
√ trust anchors are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
√ issuer cert is valid for at least 60 days
√ issuer cert is issued by the trust anchor

linkerd-webhooks-and-apisvc-tls
-------------------------------
√ tap API server has valid cert
√ tap API server cert is valid for at least 60 days
√ proxy-injector webhook has valid cert
√ proxy-injector cert is valid for at least 60 days
√ sp-validator webhook has valid cert
√ sp-validator cert is valid for at least 60 days

linkerd-api
-----------
√ control plane pods are ready
√ control plane self-check
√ [kubernetes] control plane can talk to Kubernetes
√ [prometheus] control plane can talk to Prometheus
√ tap api service is running

linkerd-version
---------------
√ can determine the latest version
√ cli is up-to-date

control-plane-version
---------------------
√ control plane is up-to-date
√ control plane and cli versions match

linkerd-prometheus
------------------
√ prometheus add-on service account exists
√ prometheus add-on config map exists
√ prometheus pod is running

linkerd-grafana
---------------
√ grafana add-on service account exists
√ grafana add-on config map exists
√ grafana pod is running

Status check results are √

```

### Environment

- Kubernetes Version: v1.18.9+k3s1
- Cluster Environment: k3d
- Host OS: "Pop!_OS 20.04 LTS" (Ubuntu 20.04)
- Linkerd version: Found on `edge-20.11.1`, also happens with `stable-2.9` and `stable-2.8.1`

### Additional context
When targeting a specific deployment or pod, the command works and the output is displayed:

```bash
linkerd tap -n linkerd deploy/linkerd-prometheus
req id=0:0 proxy=in src=10.42.0.1:47096 dst=10.42.0.17:9090 tls=no_tls_from_remote :method=GET :authority=10.42.0.17:9090 :path=/-/ready
rsp id=0:0 proxy=in src=10.42.0.1:47096 dst=10.42.0.17:9090 tls=no_tls_from_remote :status=200 latency=833µs
end id=0:0 proxy=in src=10.42.0.1:47096 dst=10.42.0.17:9090 tls=no_tls_from_remote duration=66µs response-length=21B
req id=0:1 proxy=out src=10.42.0.17:41954 dst=10.42.0.14:4191 tls=true :method=GET :authority=10.42.0.14:4191 :path=/metrics
rsp id=0:1 proxy=out src=10.42.0.17:41954 dst=10.42.0.14:4191 tls=true :status=200 latency=3033µs
end id=0:1 proxy=out src=10.42.0.17:41954 dst=10.42.0.14:4191 tls=true duration=260µs response-length=2628B
req id=0:2 proxy=out src=10.42.0.17:48764 dst=10.42.0.10:4191 tls=true :method=GET :authority=10.42.0.10:4191 :path=/metrics
rsp id=0:2 proxy=out src=10.42.0.17:48764 dst=10.42.0.10:4191 tls=true :status=200 latency=1847µs
end id=0:2 proxy=out src=10.42.0.17:48764 dst=10.42.0.10:4191 tls=true duration=46µs response-length=4002B
```

I've checked the logs from all the pods in the kube-system and linkerd and nothing is logged when the tap command fails.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the issue with k3d using the commands in the report, comparing `linkerd tap -n linkerd po --verbose` and `deploy --verbose` with a specific pod or deployment target. Trace the `linkerd tap` CLI path for category resource requests and its response stream handling. Done means category targets no longer fail with unexpected EOF while specific-resource output continues to work.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
cli, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.