Tunnels dead since 2023-10-02T22:00Z: reply 500 from https://global.rel.tunnels.api.visualstudio.com/api/v1/tunnels

Configuration

• VSCode UI, from Help/About box:
  Version: 1.82.3
  Commit: fdb98833154679dbaa7af67a5a29fe19e55c2b73
  Date: 2023-10-02T10:51:22.294Z
  Browser: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Code/1.82.3 Chrome/114.0.5735.289 Electron/25.8.1 Safari/537.36
• VSCode cli remote version: code 1.82.3 (commit fdb98833154679dbaa7af67a5a29fe19e55c2b73)
• Local OS Version: Microsoft Windows [Version 10.0.22621.2283]
• Remote OS Version:
  Linux 6.1.0-12-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.52-1 (2023-09-07) x86_64 GNU/Linux
  (headless cloud VM, Debian 12 'bookworm')
• Remote Extension/Connection Type: Tunnel

Issue manifestation

While working on a project in VSCode on Windows, connection to the host was suddenly lost. An attempt to run the command "Tunnel: Connect to tunnel..." ended up in two different ways, still reproducing:

1. The tunnels dropdown opened after the progress gofer flew for less than 1s, with the empty list of tunnels (I have used to have 2 registered tunnels) and the option "Create a tunnel."
2. The progress gofer flying for 20 minutes, at which point I closed the VSCode UI.

Cross-references

#9054
#9055
#9057

An attempt to bootstrap the remote from ground up (failed)

The local VSCode is not involved. All session logs are the remote's command line. The tunnel worked previously on the same machine, installed initially with the same steps I'm attempting to retrace here, under my user slice systemd in a locked logind session that started during power-up of the cloud instance.

All logged times are ‒0700.

$ cd
$ pwd
/home/kkm

Remove existing tunnel service, unregister the tunnel

$ systemctl --user disable --now code-tunnel.service
Removed "/home/kkm/.config/systemd/user/code-tunnel.service".
Removed "/home/kkm/.config/systemd/user/default.target.wants/code-tunnel.service".
$ code tunnel unregister --verbose
[2023-10-03 14:13:22] debug Using file keychain storage due to: user prefers file storage:
[2023-10-03 14:13:22] trace No token in keyring, getting a new one
[2023-10-03 14:13:22] debug [reqwest::connect] starting new connection: https://github.com/
To grant access to the server, please log into https://github.com/login/device and use code 22AD-8B83
[2023-10-03 14:13:27] trace refresh poll failed, retrying: Error getting authorization: authorization_pending The authorization request is still pending.
#### (... repeated 6 times while I'm fumbling in the browser with the OTP ...)
[2023-10-03 14:13:58] debug [reqwest::connect] starting new connection: https://usw3.rel.tunnels.api.visualstudio.com/
$ code tunnel status --verbose
{"tunnel":null,"service_installed":false}

Clean everything, install new minimal code cli

As directed at https://code.visualstudio.com/docs/remote/tunnels

$ curl -Lk 'https://code.visualstudio.com/sha/download?build=stable&os=cli-alpine-x64' --output vscode_cli.231003.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   133  100   133    0     0    181      0 --:--:-- --:--:-- --:--:--   181
100 7664k  100 7664k    0     0  9154k      0 --:--:-- --:--:-- --:--:-- 9154k

#### (compare with the previous download: yes, it has changed)
$ md5sum vscode*gz
9e21b5ca6e7f451d6df9ea1688dc8b9c  vscode_cli-230927.tar.gz
538a5f99b86fe209ee758e363d82f07b  vscode_cli-231003.tar.gz

$ rm ./local/bin/code
$ tar xvaf vscode_cli-231003.tar.gz -C .local/bin
-rwxrwxr-x cloudtest/cloudtest 17981560 2023-10-02 03:23 code
$ type -a code
code is /home/kkm/.local/bin/code
$ code --version
code 1.82.3 (commit fdb98833154679dbaa7af67a5a29fe19e55c2b73)

$ cat .vscode-server/data/Machine/settings.json
{
}
#### (no need to preserve machine settings; I'm only using User and Workplace configs.)
$ rm -rf .vscode/ .vscode-server/
$ code tunnel status --verbose
{"tunnel":null,"service_installed":false}

Attempt to re-register the tunnel

Failed w/500 from https://global.rel.tunnels.api.visualstudio.com/api/v1/tunnels?includePorts=true&tokenScopes=host

$ code tunnel --verbose --log trace --accept-server-license-terms
*
* Visual Studio Code Server
*
* By using the software, you agree to
* the Visual Studio Code Server License Terms (https://aka.ms/vscode-server-license) and
* the Microsoft Privacy Statement (https://privacy.microsoft.com/en-US/privacystatement).
*
[2023-10-03 14:16:10] debug Starting tunnel with `code code tunnel --verbose --log trace --accept-server-license-terms`
[2023-10-03 14:16:10] debug starting as new singleton
[2023-10-03 14:16:10] debug No code server tunnel found, creating new one
[2023-10-03 14:16:10] debug Using file keychain storage due to: error reading keyring: Platform secure storage failure: zbus error: org.freedesktop.DBus.Error.ServiceUnknown: The name org.freedesktop.secrets was not provided by any .service files
[2023-10-03 14:16:10] trace No token in keyring, getting a new one
[2023-10-03 14:16:10] debug [reqwest::connect] starting new connection: https://github.com/
To grant access to the server, please log into https://github.com/login/device and use code 94FB-4BC3
[2023-10-03 14:16:16] trace refresh poll failed, retrying: Error getting authorization: authorization_pending The authorization request is still pending.
<... repeated 5 more times while I was entering the OTP code ...>
[2023-10-03 14:16:46] debug [reqwest::connect] starting new connection: https://global.rel.tunnels.api.visualstudio.com/
✔ What would you like to call this machine? · [REDACTED-TUNNEL-NAME] (#### I accepted the default, as before.)
[2023-10-03 14:16:55] info Creating tunnel with the name: [REDACTED-TUNNEL-NAME]
[2023-10-03 14:16:55] trace Found token in keyring
[2023-10-03 14:16:55] debug [reqwest::connect] starting new connection: https://api.github.com/
[2023-10-03 14:16:57] trace Found token in keyring
(#### NB: A pause of 282 seconds in reported activity)
[2023-10-03 14:21:39] error Could not create tunnel with name: [REDACTED-TUNNEL-NAME]
Reason: ResponseError(ResponseError { url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("global.rel.tunnels.api.visualstudio.com")), port: None, path: "/api/v1/tunnels", query: Some("includePorts=true&tokenScopes=host"), fragment: None }, status_code: 500, data: Some("{\"message\":\"Server error\",\"stackTrace\":null}"), request_id: Some("0d1b17b8-e40a-4673-8f97-ca975638deab") })

Attempts to discover more logs (unsuccessful)

$ ls -lR .vscode*
.vscode:
total 4
drwxr-xr-x 2 kkm kkm 4096 2023-10-03 14:16:46 cli

.vscode/cli:
total 8
-rw-r--r-- 1 kkm kkm 142 2023-10-03 14:16:46 token.json
-rw-r--r-- 1 kkm kkm  64 2023-10-03 14:16:10 tunnel-stable.lock

$ ls -ltF $TMP
srwxr-xr-x 1 kkm kkm 0 2023-10-03 14:16:10 /home/kkm/.tmp/code-29668724-6c7f-4d75-8bbb-aefb68db082e=

$ stat /home/kkm/.tmp/code-29668724-6c7f-4d75-8bbb-aefb68db082e
  File: /home/kkm/.tmp/code-29668724-6c7f-4d75-8bbb-aefb68db082e
  Size: 0               Blocks: 0          IO Block: 4096   socket
Device: 8,1     Inode: 524321      Links: 1
Access: (0755/srwxr-xr-x)  Uid: ([GID-UID-REDACTED]/     kkm)   Gid: ([GID-UID-REDACTED]/     kkm)
Access: 2023-10-03 14:16:10.843972903 -0700
Modify: 2023-10-03 14:16:10.843972903 -0700
Change: 2023-10-03 14:16:10.843972903 -0700
 Birth: 2023-10-03 14:16:10.843972903 -0700

$ #### The VM is set up to combine journald output into a single database: no separate '--user' journals.
$ cat /etc/systemd/journald.conf{,.d/*} | grep -Pv '^\s*(#|$)'
[Journal]
[Journal]
Storage=persistent
SplitMode=none   <=== THIS
ForwardToConsole=yes
ForwardToWall=no
MaxLevelConsole=info
SystemMaxUse=2G

$ journalctl --since='14:11:00' --no-pager
-- No entries --
$ journalctl --since='14:11:00' --no-pager --user
-- No entries --