KeypairFromAgent throws fetch into an infinite loop

I'm not sure I'm using it right, but the following code throws git_remote_fetch into some sort of infinite loop in case the agent can't find an appropriate key.

import pygit2

class MyCallbacks(pygit2.RemoteCallbacks):
    def credentials(self, url, username_from_url, allowed_types):
        # print('Credentials callback called')
        return pygit2.KeypairFromAgent(username_from_url)

repo = pygit2.Repository('foo')
repo.remotes['origin'].fetch(callbacks=MyCallbacks())
print("OK")

Here's an example:

$ git clone git@<some-ssh-repository> foo
Cloning into 'foo'...
remote: Enumerating objects: 3, done.
remote: Counting objects: 100% (3/3), done.
remote: Total 3 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (3/3), done.
$ python3 test.py  # Using my regular ssh agent which has the correct key
OK
$ eval $(ssh-agent)  # Create a new ssh-agent with no key and update env variables
$ python3 test.py  # Never returns...

If I uncomment the print statement in the python snippet above, I can see that my callback is called over and over.

There's one more thing that is odd but might help: when I interrupt the script with ctrl-c, I don't always get the same traceback. In the second case bellow it seems that the "correct" exception (_pygit2.GitError: callback failed to initialize SSH credentials) is raised at some point but just doesn't reach my code:

❯❯❯ python3 test.py
# ^C
Traceback (most recent call last):
  File "test.py", line 11, in <module>
    repo.remotes['origin'].fetch(callbacks=MyCallbacks())
  File "/home/gregoire/.local/lib/python3.7/site-packages/pygit2/remote.py", line 406, in fetch
    err = C.git_remote_fetch(self._remote, arr, fetch_opts, to_bytes(message))
KeyboardInterrupt
❯❯❯ python3 test.py
# ^C
From cffi callback <function RemoteCallbacks._credentials_cb at 0x7f5e027700d0>:
Traceback (most recent call last):
  File "/home/gregoire/.local/lib/python3.7/site-packages/pygit2/remote.py", line 304, in _credentials_cb
    ccred = get_credentials(credentials, url, username, allowed)
  File "/home/gregoire/.local/lib/python3.7/site-packages/pygit2/remote.py", line 515, in get_credentials
    if pubkey is None and privkey is None:
KeyboardInterrupt
Traceback (most recent call last):
  File "test.py", line 11, in <module>
    repo.remotes['origin'].fetch(callbacks=MyCallbacks())
  File "/home/gregoire/.local/lib/python3.7/site-packages/pygit2/remote.py", line 409, in fetch
    check_error(err)
  File "/home/gregoire/.local/lib/python3.7/site-packages/pygit2/errors.py", line 64, in check_error
    raise GitError(message)
_pygit2.GitError: callback failed to initialize SSH credentials

Start with the remote.py paths shown in the traceback, especially RemoteCallbacks._credentials_cb and get_credentials. Reproduce the fetch with an ssh-agent containing no key, then trace the callback and credential error handling. Done means the failed credential lookup stops retrying and the expected GitError reaches the caller.