How to use ssh-agent from devcontainer CLI ?

I have a devcontainer that I will be using in vscode and in CI.

In the CI we use devcontainer up/exec

in vscode, we see there is a dev container helper that sets up ssh-agent.

In the CI we setup ssh-agent and add the key, but I cannot find a way to have the same devcontainer works in both environment.

I run vscode on windows and remote container are on linux. I can mount my SSH_AUTH_SOCK to /tmp/ssh-agent.socket and then I can start ssh-agent -a /tmp/ssh-agent.socket, but ssh-add -L still report no identities and outside of the container ssh-add -L do report at least one key that I did add to the ssh-agent.

Would you have an example of how one can use devcontainer with docker-compose to access the local ssh-agent, in a compatible way that would also work within vscode itself?

(when I load the same devcontainer in vscode, the SSH_AUTH_SOCK gets transformed into a ".", yet my linux remove is using ssh-agent and has a proper SSH_AUTH_SOCK value). Somehow I think the Windows version is being used, for which no SSH_AUTH_SOCK exists, but I do have the ssh-agent running on Windows.

I would definitely prefer that the remote container uses the remote linux ssh-agent than my windows.