aio-libs / aio-libs/aiohttp

Response object reference to session returns None

Abierto
#8,724 2 comentarios 0 reacciones 0 asignados Ver en GitHub
enhancement
Lenguaje dominante
Python
Estrellas
16.5k
Forks
2.4k
Merge medio
17 h 22 min
PR fusionados (30 d)
212

Descripción

### Describe the bug

I have a script that requests thousands of pages from an API at a rate of 1 req/sec due to rate limits. This causes that the session token that is created after authenticating into the API expires multiple times during execution.

To solve that implemented a custom `raise_for_status` function that refreshes a session token after getting an HTTP 500 error response code due to faulty implementation of the API that I'm consuming. Here's a barebones (and redacted) copy of the relevant script functions:

``` python
import backoff
import asyncio
import aiohttp

async def custom_raise_for_status(response: aiohttp.ClientResponse):
if response.status >= 400:
if response.status in (403, 500):
logger.error("Token has expired. Refreshing...")
await refresh_token(session=response._session)
else:
logger.error(f"Unexpected HTTP error code {response.status}")
raise aiohttp.ClientResponseError(response.request_info, response.history)

# Token header is already defined in the session.
async def refresh_token(session: aiohttp.ClientSession):
url = f"{API_URL}/auth_token/extend"
async with session.get(url) as response:
assert response.status == 200

@backoff.on_exception(
backoff.expo, aiohttp.ClientResponseError, max_tries=20, logger=logger
)
async def get_api_endpoint(session: aiohttp.ClientSession):
url = f"{API_URL}/path/to/endpoint"
async with session.get(url) as response:
data = await response.json()
# Do stuff with data

async def main():
# Do stuff
headers = {"Content-Type": "application/json", "charset": "UTF-8"}
ssl_context = ssl.create_default_context(cafile=certifi.where())
connector = aiohttp.TCPConnector(ssl=ssl_context)
async with aiohttp.ClientSession(
headers=headers, raise_for_status=custom_raise_for_status, connector=connector
) as session:
logger.info("Logging in...")
await login(session=session)
await get_api_endpoint(session=session)
# Do more stuff
```

However, the reference to `response._session` is returning a `None` value.

### To Reproduce

1. Implement the code showed previously.
2. Call to API responds with HTTP 500 after token expires.
3. An error ocurs.

### Expected behavior

1. The HTTP 500 triggers raising `ClientResponseError` exception.
2. The `backoff` annotation calls the `custom_raise_for_status` function.
3. The `response` variable uses the reference to `_session` as a parameter to refresh the session token.
4. No exception is raised and the program continues.

### Logs/tracebacks

```python-traceback
on 585: ERROR:module.container_findings:Token has expired. Refreshing...
|███████▌⚠︎ | (!) 585/3139 [19%] in 9:50.4 (0.99/s)
+ Exception Group Traceback (most recent call last):
| File "C:\path\to\project\main.py", line 115, in
| main()
| File "C:\path\to\project\main.py", line 101, in main
| count_container_findings(SCORE_CARD)
| File "C:\path\to\project\module\client.py", line 188, in count
| asyncio.run(main())
| File "C:\Users\user\AppData\Local\Programs\Python\Python312\Lib\asyncio\runners.py", line 194, in run
| return runner.run(main)
| ^^^^^^^^^^^^^^^^
| File "C:\Users\user\AppData\Local\Programs\Python\Python312\Lib\asyncio\runners.py", line 118, in run
| return self._loop.run_until_complete(task)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "C:\Users\user\AppData\Local\Programs\Python\Python312\Lib\asyncio\base_events.py", line 687, in run_until_complete
| return future.result()
| ^^^^^^^^^^^^^^^
| File "C:\path\to\project\module\client.py", line 178, in main
| hosts = await get_all_pages(session=session, pages=pages)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "C:\path\to\project\module\client.py", line 118, in get_all_pages
| async with GatheringTaskGroup() as tg:
| File "C:\Users\user\AppData\Local\Programs\Python\Python312\Lib\asyncio\taskgroups.py", line 145, in __aexit__
| raise me from None
| ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
| File "c:\path\to\project\.venv\Lib\site-packages\backoff\_async.py", line 151, in retry
| ret = await target(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "C:\path\to\project\module\client.py", line 103, in get_api_endpoint
| async with session.get(url) as response:
| File "c:\path\to\project\.venv\Lib\site-packages\aiohttp\client.py", line 1353, in __aenter__
| self._resp = await self._coro
| ^^^^^^^^^^^^^^^^
| File "c:\path\to\project\.venv\Lib\site-packages\aiohttp\client.py", line 785, in _request
| await raise_for_status(resp)
| File "C:\path\to\project\module\client.py", line 57, in custom_raise_for_status
| await refresh_token(response._session)
| File "C:\path\to\project\module\client.py", line 74, in refresh_token
| async with session.get(url) as response:
| ^^^^^^^^^^^
| AttributeError: 'NoneType' object has no attribute 'get'
+------------------------------------
```

### Python Version

```console
$ python --version
Python 3.12.4
```

### aiohttp Version

```console
$ python -m pip show aiohttp
Name: aiohttp
Version: 3.10.3
Summary: Async http client/server framework (asyncio)
Home-page: https://github.com/aio-libs/aiohttp
Author:
Author-email:
License: Apache 2
Location: c:\Users\devel\Documents\Palo Alto\dish-score-cards\.venv\Lib\site-packages
Requires: aiohappyeyeballs, aiosignal, attrs, frozenlist, multidict, yarl
Required-by:
```

### multidict Version

```console
$ python -m pip show multidict
Name: multidict
Version: 6.0.5
Summary: multidict implementation
Home-page: https://github.com/aio-libs/multidict
Author: Andrew Svetlov
Author-email: andrew.svetlov@gmail.com
License: Apache 2
Location: c:\Users\devel\Documents\Palo Alto\dish-score-cards\.venv\Lib\site-packages
Requires:
Required-by: aiohttp, yarl
```

### yarl Version

```console
$ python -m pip show yarl
Name: yarl
Version: 1.9.4
Summary: Yet another URL library
Home-page: https://github.com/aio-libs/yarl
Author: Andrew Svetlov
Author-email: andrew.svetlov@gmail.com
License: Apache-2.0
Location: c:\Users\devel\Documents\Palo Alto\dish-score-cards\.venv\Lib\site-packages
Requires: idna, multidict
Required-by: aiohttp
```

### OS

Windows 11

### Related component

Client

### Additional context

_No response_

### Code of Conduct

- [X] I agree to follow the aio-libs Code of Conduct

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.