rstudio / rstudio/helm

Adding a sidecar to session pods via `job-json-overrides` prevents Workbench from connecting to session

Open
#145 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

team: workbench
Dominant language
Markdown
Stars
46
Forks
40
Avg merge
4h 3m
Merged PRs (30d)
6

Description

We run node_exporter in user sessions to get additional metrics on session read/write behavior. We currently install it in each session container but would like to run it as a sidecar instead to simplify our images. When I added the following to the job-json-overrides section of the config:

config:
  profiles:
    launcher.kubernetes.profiles.conf:
      "*":
        job-json-overrides:
          - target: /spec/template/spec/containers/1
            json:
              image: "quay.io/prometheus/node-exporter:v1.3.1"
              name: "node-exporter"
              command:
                - node_exporter
                - --collector.disable-defaults
                - --collector.mountstats
                - --web.disable-exporter-metrics
            name: nodeExporterSidecar

the sidecar is created and the pod starts. However, Workbench is unable to connect to the pod and reports the following error in /var/log/rstudio/launcher/rstudio-kubernetes-launcher.log:

2022-01-19T01:24:36.087036Z [rstudio-kubernetes-launcher] ERROR system error 71 (Protocol error) [description: watchPodLogs: received non-streaming response: Bad Request - {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"a container name must be specified for pod session-34051cfc78a31b207e05e-USERNAME---rstudio-sessio--1-vblzj, choose one of: [5964456645756e31523730702b4b57455844483063773d3d node-exporter]","reason":"BadRequest","code":400}|||]; OCCURRED AT rstudio::job_launcher::impls::kubernetes::KubernetesApi::watchPodLogs(const string&, const OnLogOutputHandler&, const ErrorHandler&)::<lambda(const rstudio::core::http::Response&)> src/cpp/job_launcher/impls/kubernetes/KubernetesApi.cpp:1092; LOGGED FROM: rstudio::job_launcher::impls::kubernetes::KubernetesOutputStream::init()::<lambda(const rstudio::core::json::Array&)>::<lambda(const rstudio::core::Error&)> src/cpp/job_launcher/impls/kubernetes/KubernetesOutputStream.cpp:85

It looks like the Kubernetes Plugin assumes that there will only be one container per session pod. There's no issue with the Helm chart, so If this isn't the right venue for this error please let me know and I'll report it via support and our sales rep.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with src/cpp/job_launcher/impls/kubernetes/KubernetesApi.cpp:1092 and KubernetesOutputStream.cpp:85, where the log watch reports the ambiguous container response. Reproduce the issue using the shown job-json-overrides configuration with a sidecar, then verify that Workbench connects and session logs can be watched when the pod has multiple containers.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, kubernetes
Domain
infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.