Git lfs with ssh-key doesn't work on GitHub Enterprise Server

summary

Wtih GitHub Enterprise Server without domain isolation,
code is successfully downloaded, but lfs is not working with ssh.
I have checked #415 and tried to use ssh.
How to use lfs with ssh on actions/checkout ?

details

workflow.yaml is like:

 - name: checkout my_repo
        uses: actions/checkout@v2
        with:
          # private repository
          repository: 'noymer/my_repo'
          path: 'my_repo'
          ssh-key: ${{ secrets.MY_GITHUB_SSH_KEY }}
          ssh-known-hosts: ${{ env.SSH_KNOWN_HOSTS }}
          lfs: true
        env:
          GIT_TRACE: 1
          GIT_CURL_VERBOSE: 1

log is like:

15:54:44.002201 trace git-lfs: run_command: sh -c "/usr/bin/ssh" -i "$RUNNER_TEMP/19aa3905-8c76-4f98-9d3d-1590305aaca8" -o StrictHostKeyChecking=yes -o CheckHostIP=no -o "UserKnownHostsFile=$RUNNER_TEMP/19aa3905-8c76-4f98-9d3d-1590305aaca8_known_hosts" git@hoge.com 'git-lfs-authenticate noymer/my_repo.git download'
  15:54:44.002230 trace git-lfs: exec: sh '-c' '"/usr/bin/ssh" -i "$RUNNER_TEMP/19aa3905-8c76-4f98-9d3d-1590305aaca8" -o StrictHostKeyChecking=yes -o CheckHostIP=no -o "UserKnownHostsFile=$RUNNER_TEMP/19aa3905-8c76-4f98-9d3d-1590305aaca8_known_hosts" git@hoge.com 'git-lfs-authenticate noymer/my_repo.git download''
  15:54:44.313164 trace git-lfs: api: batch 22 files
  15:54:44.313385 trace git-lfs: HTTP: POST https://hoge.com/_lfs/noymer/my_repo/objects/batch
  > POST /_lfs/noymer/my_repo/objects/batch HTTP/1.1
  > Host: hoge.com
  > Accept: application/vnd.git-lfs+json; charset=utf-8
  > Authorization: RemoteAuth gitauth-v1-gnu-erCHQGDvNjNlKNTuj661p4ZtTBRbzHU_GVvUTzTpIJLOYKS4jIKmbWVtYmVysHVzZXI6OTA6eS1rYXdhYmWlcHJvdG-jc3No
  > Authorization: Basic * * * * *
  > Content-Length: 2087
  > Content-Type: application/vnd.git-lfs+json; charset=utf-8
  > User-Agent: git-lfs/2.13.3 (GitHub; linux amd64; go 1.16.2)
  > 
  {"operation":"download","objects":[{"oid":"dda5871694340eed21d00c9bc002048065e39bd2671cddd05002c91bf12580c6","size":247728428},
  ...
  {"oid":"05111001db26d788c42d2133675f44b550c9c1bfe912440b708042bebad8043d","size":109693}],"transfers":["lfs-standalone-file","basic"],"ref":{"name":"HEAD"}}15:54:44.382358 trace git-lfs: HTTP: 400
  
  
  < HTTP/1.1 400 Bad Request
  < Content-Length: 161118
  < Content-Type: text/html
  < Date: Wed, 19 May 2021 06:54:44 GMT
  < Etag: "609e5295-2755e"
  < Request-Hmac: 1621407284.9f62c8e3feeee157686198fa8c51c67a4cc9acf08b0741f8e92ead5920182d02
  < Server: GitHub.com
  < 
  15:54:44.382422 trace git-lfs: api error: Client error: https://hoge.com/_lfs/noymer/my_repo/objects/batch
  batch response: Client error: https://hoge.com/_lfs/noymer/my_repo/objects/batch
  Error: error: failed to fetch some objects from 'https://hoge.com/noymer/my_repo.git/info/lfs'

• GitHub Enterprise Server env
  • v 3.0.7
  • without subdomain isolation
  • (domain is not hoge.com , but substituted to hoge.com in log and yaml)
• Self-hosted runner environment:
  • OS: Linux, Ubuntu 18.04
  • git version 2.31.1
  • git-lfs/2.13.3 (GitHub; linux amd64; go 1.16.2)

Start with the workflow.yaml configuration using actions/checkout@v2, an SSH key, ssh-known-hosts, and lfs: true. Reproduce the Git LFS request on GitHub Enterprise Server 3.0.7 with the self-hosted Ubuntu runner and compare it with the included trace. Done means the private repository's LFS objects download successfully over SSH.