Background process 'user_directory.refresh_remote_profiles' threw an exception

### Description

I'm seeing stacktraces like the below every couple of minutes in my logs

### Steps to reproduce

? have a busy server, maybe?

### Homeserver

sw1v.org

### Synapse Version

1.113.0

### Installation Method

pip (from PyPI)

### Database

postgres

### Workers

Single process

### Platform

debian

### Configuration

_No response_

### Relevant log output

```shell
2024-08-19 10:59:54,658 - synapse.metrics.background_process_metrics - 253 - ERROR - user_directory.refresh_remote_profiles-99 - Background process 'user_directory.refresh_remote_profiles' threw an exception
Capture point (most recent call last):
File "", line 1, in
File "/home/synapse/src/synapse/synapse/app/homeserver.py", line 401, in main
run(hs)
File "/home/synapse/src/synapse/synapse/app/homeserver.py", line 380, in run
_base.start_reactor(
File "/home/synapse/src/synapse/synapse/app/_base.py", line 204, in start_reactor
run()
File "/home/synapse/src/synapse/synapse/app/_base.py", line 186, in run
run_command()
File "/home/synapse/src/synapse/synapse/app/_base.py", line 161, in
run_command: Callable[[], None] = lambda: reactor.run(),
File "/home/synapse/poetry-env/matrix-synapse-o-qzlfnx-py3.11/lib/python3.11/site-packages/twisted/internet/base.py", line 701, in run
self.mainLoop()
File "/home/synapse/poetry-env/matrix-synapse-o-qzlfnx-py3.11/lib/python3.11/site-packages/twisted/internet/base.py", line 709, in mainLoop
self.runUntilCurrent()
File "/home/synapse/poetry-env/matrix-synapse-o-qzlfnx-py3.11/lib/python3.11/site-packages/twisted/internet/base.py", line 1063, in runUntilCurrent
f(*a, **kw)
File "/home/synapse/poetry-env/matrix-synapse-o-qzlfnx-py3.11/lib/python3.11/site-packages/twisted/internet/defer.py", line 873, in callback
self._startRunCallbacks(result)
File "/home/synapse/poetry-env/matrix-synapse-o-qzlfnx-py3.11/lib/python3.11/site-packages/twisted/internet/defer.py", line 980, in _startRunCallbacks
self._runCallbacks()
File "/home/synapse/poetry-env/matrix-synapse-o-qzlfnx-py3.11/lib/python3.11/site-packages/twisted/internet/defer.py", line 1074, in _runCallbacks
current.result = callback( # type: ignore[misc]
File "/home/synapse/poetry-env/matrix-synapse-o-qzlfnx-py3.11/lib/python3.11/site-packages/twisted/internet/defer.py", line 1960, in _gotResultInlineCallbacks
_inlineCallbacks(r, gen, status, context)
File "/home/synapse/poetry-env/matrix-synapse-o-qzlfnx-py3.11/lib/python3.11/site-packages/twisted/internet/defer.py", line 2112, in _inlineCallbacks
status.deferred.callback(callbackValue)
File "/home/synapse/poetry-env/matrix-synapse-o-qzlfnx-py3.11/lib/python3.11/site-packages/twisted/internet/defer.py", line 873, in callback
self._startRunCallbacks(result)
File "/home/synapse/poetry-env/matrix-synapse-o-qzlfnx-py3.11/lib/python3.11/site-packages/twisted/internet/defer.py", line 980, in _startRunCallbacks
self._runCallbacks()
File "/home/synapse/poetry-env/matrix-synapse-o-qzlfnx-py3.11/lib/python3.11/site-packages/twisted/internet/defer.py", line 1074, in _runCallbacks
current.result = callback( # type: ignore[misc]
File "/home/synapse/poetry-env/matrix-synapse-o-qzlfnx-py3.11/lib/python3.11/site-packages/twisted/internet/defer.py", line 873, in callback
self._startRunCallbacks(result)
File "/home/synapse/poetry-env/matrix-synapse-o-qzlfnx-py3.11/lib/python3.11/site-packages/twisted/internet/defer.py", line 980, in _startRunCallbacks
self._runCallbacks()
File "/home/synapse/poetry-env/matrix-synapse-o-qzlfnx-py3.11/lib/python3.11/site-packages/twisted/internet/defer.py", line 1074, in _runCallbacks
current.result = callback( # type: ignore[misc]
File "/home/synapse/poetry-env/matrix-synapse-o-qzlfnx-py3.11/lib/python3.11/site-packages/twisted/internet/defer.py", line 1960, in _gotResultInlineCallbacks
_inlineCallbacks(r, gen, status, context)
File "/home/synapse/poetry-env/matrix-synapse-o-qzlfnx-py3.11/lib/python3.11/site-packages/twisted/internet/defer.py", line 2014, in _inlineCallbacks
result = context.run(gen.send, result)
Traceback (most recent call last):
File "/home/synapse/src/synapse/synapse/metrics/background_process_metrics.py", line 251, in run
return await func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/synapse/src/synapse/synapse/handlers/user_directory.py", line 593, in process
await self._unsafe_refresh_remote_profiles()
File "/home/synapse/src/synapse/synapse/handlers/user_directory.py", line 642, in _unsafe_refresh_remote_profiles
self._refresh_remote_profiles_call_later = self.clock.call_later(
^^^^^^^^^^^^^^^^^^^^^^
File "/home/synapse/src/synapse/synapse/util/__init__.py", line 204, in call_later
return self._reactor.callLater(delay, wrapped_callback, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/synapse/poetry-env/matrix-synapse-o-qzlfnx-py3.11/lib/python3.11/site-packages/twisted/internet/base.py", line 968, in callLater
assert delay >= 0, f"{delay} is not greater than or equal to 0 seconds"
^^^^^^^^^^
AssertionError: -13 is not greater than or equal to 0 seconds
```

### Anything else that would be useful to know?

installed with poetry