Adding a sidecar to session pods via `job-json-overrides` prevents Workbench from connecting to session
Nobody has claimed this yet.
- Dominant language
- Markdown
- Stars
- 46
- Forks
- 40
- Avg merge
- 4h 3m
- Merged PRs (30d)
- 6
Description
We run node_exporter in user sessions to get additional metrics on session read/write behavior. We currently install it in each session container but would like to run it as a sidecar instead to simplify our images. When I added the following to the job-json-overrides section of the config:
config:
profiles:
launcher.kubernetes.profiles.conf:
"*":
job-json-overrides:
- target: /spec/template/spec/containers/1
json:
image: "quay.io/prometheus/node-exporter:v1.3.1"
name: "node-exporter"
command:
- node_exporter
- --collector.disable-defaults
- --collector.mountstats
- --web.disable-exporter-metrics
name: nodeExporterSidecar
the sidecar is created and the pod starts. However, Workbench is unable to connect to the pod and reports the following error in /var/log/rstudio/launcher/rstudio-kubernetes-launcher.log:
2022-01-19T01:24:36.087036Z [rstudio-kubernetes-launcher] ERROR system error 71 (Protocol error) [description: watchPodLogs: received non-streaming response: Bad Request - {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"a container name must be specified for pod session-34051cfc78a31b207e05e-USERNAME---rstudio-sessio--1-vblzj, choose one of: [5964456645756e31523730702b4b57455844483063773d3d node-exporter]","reason":"BadRequest","code":400}|||]; OCCURRED AT rstudio::job_launcher::impls::kubernetes::KubernetesApi::watchPodLogs(const string&, const OnLogOutputHandler&, const ErrorHandler&)::<lambda(const rstudio::core::http::Response&)> src/cpp/job_launcher/impls/kubernetes/KubernetesApi.cpp:1092; LOGGED FROM: rstudio::job_launcher::impls::kubernetes::KubernetesOutputStream::init()::<lambda(const rstudio::core::json::Array&)>::<lambda(const rstudio::core::Error&)> src/cpp/job_launcher/impls/kubernetes/KubernetesOutputStream.cpp:85
It looks like the Kubernetes Plugin assumes that there will only be one container per session pod. There's no issue with the Helm chart, so If this isn't the right venue for this error please let me know and I'll report it via support and our sales rep.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with src/cpp/job_launcher/impls/kubernetes/KubernetesApi.cpp:1092 and KubernetesOutputStream.cpp:85, where the log watch reports the ambiguous container response. Reproduce the issue using the shown job-json-overrides configuration with a sidecar, then verify that Workbench connects and session logs can be watched when the pod has multiple containers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, kubernetes
- Domain
- infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100