EOF when running linkerd tap command against a category of resources on a k3d cluster
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 11.5k
- Forks
- 1.4k
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 53
Description
### What is the issue?
Running the `linkerd tap` command against a control plane deployed to a k3d cluster results in an EOF error when targeting an entire category of kubernetes resources (e.g. Pod or Deployment).
The same command run against a specific resource (deploy/linkerd-prometheus, or a specific pod) works as expected.
### How can it be reproduced?
- Provision a cluster using [k3d](https://k3d.io/) (The number of nodes doesn't matter)
-- `k3d cluster create tap-test`
- Deploy the linkerd control plane
-- `linkerd install | kubectl apply -f -`
- Run tap against all resource types in the linkerd control plane
-- Pods: `linkerd tap -n linkerd po --verbose`
-- Deployments: `linkerd tap -n linkerd deploy --verbose`
### Logs, error output, etc
```
DEBU[0000] Response from [https://0.0.0.0:43867/apis/tap.linkerd.io/v1alpha1/watch/namespaces/linkerd/pods//tap] had headers: map[Cache-Control:[no-cache, private] Content-Length:[205] Content-Type:[application/json] Date:[Thu, 12 Nov 2020 17:58:32 GMT]]
DEBU[0000] Waiting for data...
error reading byte stream header: error while reading bytes from message: unexpected EOF
```
#### `linkerd check` output
```text
kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API
kubernetes-version
------------------
√ is running the minimum Kubernetes API version
√ is running the minimum kubectl version
linkerd-existence
-----------------
√ 'linkerd-config' config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
√ controller pod is running
√ can initialize the client
√ can query the control plane API
linkerd-config
--------------
√ control plane Namespace exists
√ control plane ClusterRoles exist
√ control plane ClusterRoleBindings exist
√ control plane ServiceAccounts exist
√ control plane CustomResourceDefinitions exist
√ control plane MutatingWebhookConfigurations exist
√ control plane ValidatingWebhookConfigurations exist
√ control plane PodSecurityPolicies exist
linkerd-identity
----------------
√ certificate config is valid
√ trust anchors are using supported crypto algorithm
√ trust anchors are within their validity period
√ trust anchors are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
√ issuer cert is valid for at least 60 days
√ issuer cert is issued by the trust anchor
linkerd-webhooks-and-apisvc-tls
-------------------------------
√ tap API server has valid cert
√ tap API server cert is valid for at least 60 days
√ proxy-injector webhook has valid cert
√ proxy-injector cert is valid for at least 60 days
√ sp-validator webhook has valid cert
√ sp-validator cert is valid for at least 60 days
linkerd-api
-----------
√ control plane pods are ready
√ control plane self-check
√ [kubernetes] control plane can talk to Kubernetes
√ [prometheus] control plane can talk to Prometheus
√ tap api service is running
linkerd-version
---------------
√ can determine the latest version
√ cli is up-to-date
control-plane-version
---------------------
√ control plane is up-to-date
√ control plane and cli versions match
linkerd-prometheus
------------------
√ prometheus add-on service account exists
√ prometheus add-on config map exists
√ prometheus pod is running
linkerd-grafana
---------------
√ grafana add-on service account exists
√ grafana add-on config map exists
√ grafana pod is running
Status check results are √
```
### Environment
- Kubernetes Version: v1.18.9+k3s1
- Cluster Environment: k3d
- Host OS: "Pop!_OS 20.04 LTS" (Ubuntu 20.04)
- Linkerd version: Found on `edge-20.11.1`, also happens with `stable-2.9` and `stable-2.8.1`
### Additional context
When targeting a specific deployment or pod, the command works and the output is displayed:
```bash
linkerd tap -n linkerd deploy/linkerd-prometheus
req id=0:0 proxy=in src=10.42.0.1:47096 dst=10.42.0.17:9090 tls=no_tls_from_remote :method=GET :authority=10.42.0.17:9090 :path=/-/ready
rsp id=0:0 proxy=in src=10.42.0.1:47096 dst=10.42.0.17:9090 tls=no_tls_from_remote :status=200 latency=833µs
end id=0:0 proxy=in src=10.42.0.1:47096 dst=10.42.0.17:9090 tls=no_tls_from_remote duration=66µs response-length=21B
req id=0:1 proxy=out src=10.42.0.17:41954 dst=10.42.0.14:4191 tls=true :method=GET :authority=10.42.0.14:4191 :path=/metrics
rsp id=0:1 proxy=out src=10.42.0.17:41954 dst=10.42.0.14:4191 tls=true :status=200 latency=3033µs
end id=0:1 proxy=out src=10.42.0.17:41954 dst=10.42.0.14:4191 tls=true duration=260µs response-length=2628B
req id=0:2 proxy=out src=10.42.0.17:48764 dst=10.42.0.10:4191 tls=true :method=GET :authority=10.42.0.10:4191 :path=/metrics
rsp id=0:2 proxy=out src=10.42.0.17:48764 dst=10.42.0.10:4191 tls=true :status=200 latency=1847µs
end id=0:2 proxy=out src=10.42.0.17:48764 dst=10.42.0.10:4191 tls=true duration=46µs response-length=4002B
```
I've checked the logs from all the pods in the kube-system and linkerd and nothing is logged when the tap command fails.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the issue with k3d using the commands in the report, comparing `linkerd tap -n linkerd po --verbose` and `deploy --verbose` with a specific pod or deployment target. Trace the `linkerd tap` CLI path for category resource requests and its response stream handling. Done means category targets no longer fail with unexpected EOF while specific-resource output continues to work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- cli, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100