tailscale / tailscale/tailscale

Tailscale SSH does not work on Google Cloud Run, `setgroups` fails

Open
#8,394 6 comments 2 reactions 0 assignees View on GitHub
bug containers L2 Few P2 Aggravating ssh T5 Usability
Dominant language
Go
Stars
36.5k
Forks
3.2k
Avg merge
2d 3h
Merged PRs (30d)
123

Description

### What is the issue?

I tried to open ssh connection from my own computer (Ubuntu 22.04 LTS) to a container running Tailscale on Google Cloud Run service but connection failed with "operation not permitted" error message.

```
masa@masa-x1$ ssh www@100.127.204.106
The authenticity of host '100.127.204.106 (100.127.204.106)' can't be established.
ED25519 key fingerprint is SHA256:nXser+W1F4BwyF12llmr2OsTh78jYJa8zZDV3WP9W4M.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '100.127.204.106' (ED25519) to the list of known hosts.
operation not permitted
Connection to 100.127.204.106 closed.
```

From Google Cloud Run logs I can see that the following command exited with code 1:
```
starting pty command: [/usr/sbin/tailscaled be-child ssh --uid=1000 --gid=1000 --groups=1000 --local-user=www --remote-user=mattirantakomi@github --remote-ip=100.111.215.88 --has-tty=true --tty-name=pts/0 --shell --login-cmd=/usr/bin/login --cmd=/bin/bash -- -l]
```

I tried to debug with strace and found out that the failing syscall is setgroups.

```
...

geteuid() = 1000
geteuid() = 1000
geteuid() = 1000
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=185, si_uid=1000} ---
rt_sigreturn({mask=[]}) = 1000
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=185, si_uid=1000} ---
rt_sigreturn({mask=[]}) = 1000
getegid() = 1000
futex(0xc000060548, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x15fbd08, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
setgroups(1, [1000]) = -1 EPERM (Operation not permitted)
futex(0x15fc0c0, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xc000060548, FUTEX_WAKE_PRIVATE, 1) = 1
geteuid() = 1000
getgroups(0, NULL) = 0
write(2, "operation not permitted\n", 24operation not permitted
) = 24
exit_group(1) = ?
+++ exited with 1 +++
```

As far as I know this bug should have been already fixed on https://github.com/tailscale/tailscale/pull/6904 but I'm not sure about it.

### Steps to reproduce

- Get tailscale auth key from Tailscale Admin Web UI -> Settings -> Keys -> Auth keys -> Generate auth key (https://login.tailscale.com/admin/settings/keys). Use generated key as Cloud Run environment variable `TAILSCALE_AUTHKEY`.

- Run test container (https://github.com/mattirantakomi/cloudrun-nginx-test) on Google Cloud Run. Nginx is included and responding on port 8080 just to make Cloud Run health check happy.

```
git clone git@github.com:mattirantakomi/cloudrun-nginx-test.git
gcloud run deploy cloudrun-nginx-test \
--project \
--region europe-north1 \
--service-account \
--set-env-vars TAILSCALE_AUTHKEY="authkey-1234" \
--execution-environment gen2 \
--min-instances=1 --max-instances=1 \
--source .
```

I tried both gen1 and gen2 execution environments, neither doesn't work.

- Try to SSH to that container `ssh www@100.11.22.33`.

### Are there any recent changes that introduced the issue?

-

### OS

Linux

### OS version

Ubuntu 22.04 LTS

### Tailscale version

1.42.0

### Other software

_No response_

### Bug report

BUG-0937c81c9c29b37dc74221ea0bec337231bfcf36999b0a0ff752edde9aa2d8f2-20230621061253Z-1f619bfeaef05cc7

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.