aiopg times out while attempting to connect if falling back to an IPv4 address

### Describe the bug

While attempting to connect to CockroachDB, `aiopg` raises `TimeoutError`.

### To Reproduce

1. Run CockroachDB on `localhost`. I am running v22.2.19 on Linux.
2. Execute the minimal example below.

```python
import argparse
import asyncio
import logging
from pathlib import Path

import aiopg

async def test_db_pool(dbname: str, username: str, certs: Path):
logging.info('Creating test connection.')
kwargs = {
'host': 'localhost',
'port': 26257,
'dbname': dbname,
'user': username,
'sslmode': 'verify-full',
'sslcert': certs / f'client.{username}.crt',
'sslkey': certs / f'client.{username}.key',
'sslrootcert': certs / 'ca.crt',
}
async with aiopg.connect(**kwargs) as pool:
logging.info('Connection ready.')

def main():
logging.basicConfig(level=logging.DEBUG)
parser = argparse.ArgumentParser()
parser.add_argument('--certs-dir', action='store')
args = parser.parse_args()

asyncio.run(test_db_pool('postgres', 'root', Path(args.certs_dir)))

if __name__ == '__main__':
main()
```

### Expected behavior

The example exits successfully.

### Logs/tracebacks

```python-traceback
DEBUG:asyncio:Using selector: EpollSelector
INFO:root:Creating test connection.
Traceback (most recent call last):
File "/usr/lib/python3.11/asyncio/tasks.py", line 490, in wait_for
return fut.result()
^^^^^^^^^^^^
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/code/example/venv/lib/python3.11/site-packages/aiopg/connection.py", line 881, in _poll
await asyncio.wait_for(self._waiter, timeout)
File "/usr/lib/python3.11/asyncio/tasks.py", line 492, in wait_for
raise exceptions.TimeoutError() from exc
TimeoutError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/code/example/./aio_fail.py", line 35, in
main()
File "/code/example/./aio_fail.py", line 31, in main
asyncio.run(test_db_pool('postgres', 'root', Path(args.certs_dir)))
File "/usr/lib/python3.11/asyncio/runners.py", line 190, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "/code/example/./aio_fail.py", line 21, in test_db_pool
async with aiopg.connect(**kwargs) as pool:
File "/code/example/venv/lib/python3.11/site-packages/aiopg/utils.py", line 82, in __aenter__
self._obj = await self._coro
^^^^^^^^^^^^^^^^
File "/code/example/venv/lib/python3.11/site-packages/aiopg/connection.py", line 1225, in _connect
await self._poll(self._waiter, self._timeout) # type: ignore
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/code/example/venv/lib/python3.11/site-packages/aiopg/connection.py", line 883, in _poll
await asyncio.shield(self.close())
^^^^^^^^^^^^
File "/code/example/venv/lib/python3.11/site-packages/aiopg/connection.py", line 995, in close
self._close()
File "/code/example/venv/lib/python3.11/site-packages/aiopg/connection.py", line 977, in _close
self._loop.remove_reader(self._fileno)
File "/usr/lib/python3.11/asyncio/selector_events.py", line 345, in remove_reader
return self._remove_reader(fd)
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/asyncio/selector_events.py", line 290, in _remove_reader
self._selector.modify(fd, mask, (None, writer))
File "/usr/lib/python3.11/selectors.py", line 389, in modify
self._selector.modify(key.fd, selector_events)
FileNotFoundError: [Errno 2] No such file or directory
```

### Python Version

```console
$ python --version
Python 3.11.5
```

### aiopg Version

```console
$ python -m pip show aiopg
Name: aiopg
Version: 1.4.0
Summary: Postgres integration with asyncio.
Home-page: https://aiopg.readthedocs.io
Author: Andrew Svetlov
Author-email: andrew.svetlov@gmail.com
License: BSD
Location: /home/royiv/code/groceries/venv/lib/python3.11/site-packages
Requires: async-timeout, psycopg2-binary
Required-by:
```

### OS

Arch Linux

### Additional context

Cockroach is a distributed database that uses the PostgreSQL wire protocol for queries. `psql` is capable of connecting to it, and executing queries. (So is `psycopg2`.)

When running the example, there is a long pause after `Creating test connection.`, approximately 60s.

`aiopg` worked a few CockroachDB versions prior to this one, but I am sorry, I did not record the working version's number.

A packet captures shows `aiopg` makes the connection (the 3-way handshake happens) but no data is ever transmitted. This is a bit surprising, since at this point I am not sure what Cockroach could have done to have acted differently from vanilla PG.

### Code of Conduct

- [X] I agree to follow the aio-libs Code of Conduct