aio-libs / aio-libs/aiohttp

Response object reference to session returns None

Open
#8,724 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
16.5k
Forks
2.4k
Avg merge
17h 22m
Merged PRs (30d)
212

Description

### Describe the bug

I have a script that requests thousands of pages from an API at a rate of 1 req/sec due to rate limits. This causes that the session token that is created after authenticating into the API expires multiple times during execution.

To solve that implemented a custom `raise_for_status` function that refreshes a session token after getting an HTTP 500 error response code due to faulty implementation of the API that I'm consuming. Here's a barebones (and redacted) copy of the relevant script functions:

``` python
import backoff
import asyncio
import aiohttp

async def custom_raise_for_status(response: aiohttp.ClientResponse):
if response.status >= 400:
if response.status in (403, 500):
logger.error("Token has expired. Refreshing...")
await refresh_token(session=response._session)
else:
logger.error(f"Unexpected HTTP error code {response.status}")
raise aiohttp.ClientResponseError(response.request_info, response.history)

# Token header is already defined in the session.
async def refresh_token(session: aiohttp.ClientSession):
url = f"{API_URL}/auth_token/extend"
async with session.get(url) as response:
assert response.status == 200

@backoff.on_exception(
backoff.expo, aiohttp.ClientResponseError, max_tries=20, logger=logger
)
async def get_api_endpoint(session: aiohttp.ClientSession):
url = f"{API_URL}/path/to/endpoint"
async with session.get(url) as response:
data = await response.json()
# Do stuff with data

async def main():
# Do stuff
headers = {"Content-Type": "application/json", "charset": "UTF-8"}
ssl_context = ssl.create_default_context(cafile=certifi.where())
connector = aiohttp.TCPConnector(ssl=ssl_context)
async with aiohttp.ClientSession(
headers=headers, raise_for_status=custom_raise_for_status, connector=connector
) as session:
logger.info("Logging in...")
await login(session=session)
await get_api_endpoint(session=session)
# Do more stuff
```

However, the reference to `response._session` is returning a `None` value.

### To Reproduce

1. Implement the code showed previously.
2. Call to API responds with HTTP 500 after token expires.
3. An error ocurs.

### Expected behavior

1. The HTTP 500 triggers raising `ClientResponseError` exception.
2. The `backoff` annotation calls the `custom_raise_for_status` function.
3. The `response` variable uses the reference to `_session` as a parameter to refresh the session token.
4. No exception is raised and the program continues.

### Logs/tracebacks

```python-traceback
on 585: ERROR:module.container_findings:Token has expired. Refreshing...
|███████▌⚠︎ | (!) 585/3139 [19%] in 9:50.4 (0.99/s)
+ Exception Group Traceback (most recent call last):
| File "C:\path\to\project\main.py", line 115, in
| main()
| File "C:\path\to\project\main.py", line 101, in main
| count_container_findings(SCORE_CARD)
| File "C:\path\to\project\module\client.py", line 188, in count
| asyncio.run(main())
| File "C:\Users\user\AppData\Local\Programs\Python\Python312\Lib\asyncio\runners.py", line 194, in run
| return runner.run(main)
| ^^^^^^^^^^^^^^^^
| File "C:\Users\user\AppData\Local\Programs\Python\Python312\Lib\asyncio\runners.py", line 118, in run
| return self._loop.run_until_complete(task)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "C:\Users\user\AppData\Local\Programs\Python\Python312\Lib\asyncio\base_events.py", line 687, in run_until_complete
| return future.result()
| ^^^^^^^^^^^^^^^
| File "C:\path\to\project\module\client.py", line 178, in main
| hosts = await get_all_pages(session=session, pages=pages)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "C:\path\to\project\module\client.py", line 118, in get_all_pages
| async with GatheringTaskGroup() as tg:
| File "C:\Users\user\AppData\Local\Programs\Python\Python312\Lib\asyncio\taskgroups.py", line 145, in __aexit__
| raise me from None
| ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
| File "c:\path\to\project\.venv\Lib\site-packages\backoff\_async.py", line 151, in retry
| ret = await target(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "C:\path\to\project\module\client.py", line 103, in get_api_endpoint
| async with session.get(url) as response:
| File "c:\path\to\project\.venv\Lib\site-packages\aiohttp\client.py", line 1353, in __aenter__
| self._resp = await self._coro
| ^^^^^^^^^^^^^^^^
| File "c:\path\to\project\.venv\Lib\site-packages\aiohttp\client.py", line 785, in _request
| await raise_for_status(resp)
| File "C:\path\to\project\module\client.py", line 57, in custom_raise_for_status
| await refresh_token(response._session)
| File "C:\path\to\project\module\client.py", line 74, in refresh_token
| async with session.get(url) as response:
| ^^^^^^^^^^^
| AttributeError: 'NoneType' object has no attribute 'get'
+------------------------------------
```

### Python Version

```console
$ python --version
Python 3.12.4
```

### aiohttp Version

```console
$ python -m pip show aiohttp
Name: aiohttp
Version: 3.10.3
Summary: Async http client/server framework (asyncio)
Home-page: https://github.com/aio-libs/aiohttp
Author:
Author-email:
License: Apache 2
Location: c:\Users\devel\Documents\Palo Alto\dish-score-cards\.venv\Lib\site-packages
Requires: aiohappyeyeballs, aiosignal, attrs, frozenlist, multidict, yarl
Required-by:
```

### multidict Version

```console
$ python -m pip show multidict
Name: multidict
Version: 6.0.5
Summary: multidict implementation
Home-page: https://github.com/aio-libs/multidict
Author: Andrew Svetlov
Author-email: andrew.svetlov@gmail.com
License: Apache 2
Location: c:\Users\devel\Documents\Palo Alto\dish-score-cards\.venv\Lib\site-packages
Requires:
Required-by: aiohttp, yarl
```

### yarl Version

```console
$ python -m pip show yarl
Name: yarl
Version: 1.9.4
Summary: Yet another URL library
Home-page: https://github.com/aio-libs/yarl
Author: Andrew Svetlov
Author-email: andrew.svetlov@gmail.com
License: Apache-2.0
Location: c:\Users\devel\Documents\Palo Alto\dish-score-cards\.venv\Lib\site-packages
Requires: idna, multidict
Required-by: aiohttp
```

### OS

Windows 11

### Related component

Client

### Additional context

_No response_

### Code of Conduct

- [X] I agree to follow the aio-libs Code of Conduct

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.