rstudio / rstudio/helm

[Workbench] Server pods on arm64 don't start properly

Open
#454 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

team: workbench
Dominant language
Markdown
Stars
46
Forks
40
Avg merge
4h 3m
Merged PRs (30d)
6

Description

(Apologies for posting here as it might not be directly a helm-chart issue but we only encounter the issue in k8s environments)

Background

We're building and using arm64 images since quite some time.
Recently, we aimed to get the WB server pod running in a mixed-arch fashion on a cluster.

We are building the images using buildx in a multi-arch way and can hence ensure that both architectures have exactly the same content.
Running the images on a VM with docker works just fine, i.e. startup works and the WB is usable.
However, for an unknown reason, the arm64 pods always get stuck during startup when using it in k8s (the amd64 counterparts work just fine).

Logs

With debug logging:

 2024-01-09T09:25:33.824831Z [rserver] INFO Reading OpenID secret from '/mnt/secret-configmap/rstudio/openid-client-secret'                                                                                     │
│ 2024-01-09T09:25:36.984354Z [rstudio-kubernetes-launcher, log-source: Kubernetes] DEBUG Queueing response message [msg: {"messageType":0,"requestId":0,"responseId":0}]                                        │
│ 2024-01-09T09:25:36.984501Z [rstudio-launcher] DEBUG Received message from plugin [plugin: Kubernetes, msg: {"messageType":0,"requestId":0,"responseId":0}]                                                    │
│ 2024-01-09T09:25:36.984749Z [rstudio-launcher] DEBUG Sending message to plugin [msg: {"messageType":0,"requestId":0}, plugin: Kubernetes] 

The startup via /usr/lib/rstudio-server/bin/rserver --server-daemonize 0 >/dev/stderr always stops at Reading OpenID secret from '/mnt/secret-configmap/rstudio/openid-client-secret' while the amd64 pods continue after this point with

│ 2024-01-09T09:27:06.581937Z [rserver] INFO Reading OpenID secret from '/mnt/secret-configmap/rstudio/openid-client-secret'                                                                                     │
│ 2024-01-09T09:27:06.587874Z [rserver] INFO rserver-openid OpenID Version 0.6.0                                                                                                                                 │
│ 2024-01-09T09:27:06.588190Z [rserver] INFO Reading server user profiles from '/mnt/configmap/rstudio/profiles'                                                                                                 │
│ 2024-01-09T09:27:06.588312Z [rserver] DEBUG PyVersionsScanner: /usr/bin/python does not exist                                                                                                                  │
│ 2024-01-09T09:27:06.593626Z [rserver] INFO PyVersionsScanner: Found Python version 3.10.12 at /usr/bin/python3   

Is there any way to debug what might be happening between Reading OpenID secret and rserver-openid OpenID Version (which is never reached in the arm64 pod)?

Details

Interesting side fact: starting the same images outside k8s via docker works just fine, letting us assume that the "normal" startup is OK.

  • Workbench 2023.12.0+369.pro3
  • Ubuntu Jammy & Alma Linux 9

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the /usr/lib/rstudio-server/bin/rserver --server-daemonize 0 entry point and compare the arm64 and amd64 Kubernetes pod logs around reading /mnt/secret-configmap/rstudio/openid-client-secret. Reproduce the startup outside and inside Kubernetes, then determine why arm64 does not reach rserver-openid OpenID Version 0.6.0; done means the cause and a fix or clear diagnostic are documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, helm, kubernetes
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.