unknown service moby.sshforward.v1.SSH when building an image from github using buildkit

Description

It seems related to https://github.com/moby/buildkit/issues/2040, it was supposedly fixed by https://github.com/docker/cli/pull/3050, but it keeps happening in my Docker for mac 4.17.0 install.

Reproduce

DOCKER_BUILDKIT=1 docker build git@github.com:dockersamples/linux_tweet_app.git

This fails with the error: failed to solve with frontend dockerfile.v0: failed to read dockerfile: failed to load cache key: rpc error: code = Unimplemented desc = unknown service moby.sshforward.v1.SSH

but this succeeds:

DOCKER_BUILDKIT=0 docker build git@github.com:dockersamples/linux_tweet_app.git

Expected behavior

I expect build to pull the dockerfile from the github repository, read it and build it.

docker version

Client:
 Cloud integration: v1.0.31
 Version:           20.10.23
 API version:       1.41
 Go version:        go1.18.10
 Git commit:        7155243
 Built:             Thu Jan 19 17:35:19 2023
 OS/Arch:           darwin/arm64
 Context:           desktop-linux
 Experimental:      true

Server: Docker Desktop 4.17.0 (99724)
 Engine:
  Version:          20.10.23
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.10
  Git commit:       6051f14
  Built:            Thu Jan 19 17:31:28 2023
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.18
  GitCommit:        2456e983eb9e37e47538f59ea18f2043c9a73640
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.10.3)
  compose: Docker Compose (Docker Inc., v2.15.1)
  dev: Docker Dev Environments (Docker Inc., v0.1.0)
  extension: Manages Docker extensions (Docker Inc., v0.2.18)
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc., 0.6.0)
  scan: Docker Scan (Docker Inc., v0.25.0)
  scout: Command line tool for Docker Scout (Docker Inc., v0.6.0)

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 33
 Server Version: 20.10.23
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2456e983eb9e37e47538f59ea18f2043c9a73640
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.15.49-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 6
 Total Memory: 11.68GiB
 Name: docker-desktop
 ID: 6BJL:CA3W:QZ4J:GAGE:KXA5:S2PN:PDMH:YHVO:OYPQ:K537:G65I:2QQ2
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5000
  127.0.0.0/8
 Live Restore Enabled: false

Additional Info

I tried restoring to factory defaults, and cloning many different images, but I just can't make it work with DOCKER_BUILDKIT=1 docker build.

Using command DOCKER_BUILDKIT=1 docker buildx build git@github.com:dockersamples/linux_tweet_app.git works and builds the image correctly, but I can not see it in the docker for mac images dashboard (it does show up in docker image ls)

Start by reproducing the two commands in the issue: docker build with DOCKER_BUILDKIT=1 and docker buildx build against the sample GitHub repository. Compare the Docker CLI and BuildKit SSH-forwarding paths, using the linked BuildKit issue and CLI pull request as context; done means the regular BuildKit build reads the remote Dockerfile and completes successfully.