aio-libs / aio-libs/aiohttp

Response object reference to session returns None

Đang mở
#8,724 2 bình luận 0 reaction 0 người được giao Xem trên GitHub
enhancement
Ngôn ngữ chính
Python
Star
16.5k
Fork
2.4k
Merge trung bình
17 giờ 22 phút
Pull request đã merge (30 ngày)
212

Mô tả

### Describe the bug

I have a script that requests thousands of pages from an API at a rate of 1 req/sec due to rate limits. This causes that the session token that is created after authenticating into the API expires multiple times during execution.

To solve that implemented a custom `raise_for_status` function that refreshes a session token after getting an HTTP 500 error response code due to faulty implementation of the API that I'm consuming. Here's a barebones (and redacted) copy of the relevant script functions:

``` python
import backoff
import asyncio
import aiohttp

async def custom_raise_for_status(response: aiohttp.ClientResponse):
if response.status >= 400:
if response.status in (403, 500):
logger.error("Token has expired. Refreshing...")
await refresh_token(session=response._session)
else:
logger.error(f"Unexpected HTTP error code {response.status}")
raise aiohttp.ClientResponseError(response.request_info, response.history)

# Token header is already defined in the session.
async def refresh_token(session: aiohttp.ClientSession):
url = f"{API_URL}/auth_token/extend"
async with session.get(url) as response:
assert response.status == 200

@backoff.on_exception(
backoff.expo, aiohttp.ClientResponseError, max_tries=20, logger=logger
)
async def get_api_endpoint(session: aiohttp.ClientSession):
url = f"{API_URL}/path/to/endpoint"
async with session.get(url) as response:
data = await response.json()
# Do stuff with data

async def main():
# Do stuff
headers = {"Content-Type": "application/json", "charset": "UTF-8"}
ssl_context = ssl.create_default_context(cafile=certifi.where())
connector = aiohttp.TCPConnector(ssl=ssl_context)
async with aiohttp.ClientSession(
headers=headers, raise_for_status=custom_raise_for_status, connector=connector
) as session:
logger.info("Logging in...")
await login(session=session)
await get_api_endpoint(session=session)
# Do more stuff
```

However, the reference to `response._session` is returning a `None` value.

### To Reproduce

1. Implement the code showed previously.
2. Call to API responds with HTTP 500 after token expires.
3. An error ocurs.

### Expected behavior

1. The HTTP 500 triggers raising `ClientResponseError` exception.
2. The `backoff` annotation calls the `custom_raise_for_status` function.
3. The `response` variable uses the reference to `_session` as a parameter to refresh the session token.
4. No exception is raised and the program continues.

### Logs/tracebacks

```python-traceback
on 585: ERROR:module.container_findings:Token has expired. Refreshing...
|███████▌⚠︎ | (!) 585/3139 [19%] in 9:50.4 (0.99/s)
+ Exception Group Traceback (most recent call last):
| File "C:\path\to\project\main.py", line 115, in
| main()
| File "C:\path\to\project\main.py", line 101, in main
| count_container_findings(SCORE_CARD)
| File "C:\path\to\project\module\client.py", line 188, in count
| asyncio.run(main())
| File "C:\Users\user\AppData\Local\Programs\Python\Python312\Lib\asyncio\runners.py", line 194, in run
| return runner.run(main)
| ^^^^^^^^^^^^^^^^
| File "C:\Users\user\AppData\Local\Programs\Python\Python312\Lib\asyncio\runners.py", line 118, in run
| return self._loop.run_until_complete(task)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "C:\Users\user\AppData\Local\Programs\Python\Python312\Lib\asyncio\base_events.py", line 687, in run_until_complete
| return future.result()
| ^^^^^^^^^^^^^^^
| File "C:\path\to\project\module\client.py", line 178, in main
| hosts = await get_all_pages(session=session, pages=pages)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "C:\path\to\project\module\client.py", line 118, in get_all_pages
| async with GatheringTaskGroup() as tg:
| File "C:\Users\user\AppData\Local\Programs\Python\Python312\Lib\asyncio\taskgroups.py", line 145, in __aexit__
| raise me from None
| ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
| File "c:\path\to\project\.venv\Lib\site-packages\backoff\_async.py", line 151, in retry
| ret = await target(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "C:\path\to\project\module\client.py", line 103, in get_api_endpoint
| async with session.get(url) as response:
| File "c:\path\to\project\.venv\Lib\site-packages\aiohttp\client.py", line 1353, in __aenter__
| self._resp = await self._coro
| ^^^^^^^^^^^^^^^^
| File "c:\path\to\project\.venv\Lib\site-packages\aiohttp\client.py", line 785, in _request
| await raise_for_status(resp)
| File "C:\path\to\project\module\client.py", line 57, in custom_raise_for_status
| await refresh_token(response._session)
| File "C:\path\to\project\module\client.py", line 74, in refresh_token
| async with session.get(url) as response:
| ^^^^^^^^^^^
| AttributeError: 'NoneType' object has no attribute 'get'
+------------------------------------
```

### Python Version

```console
$ python --version
Python 3.12.4
```

### aiohttp Version

```console
$ python -m pip show aiohttp
Name: aiohttp
Version: 3.10.3
Summary: Async http client/server framework (asyncio)
Home-page: https://github.com/aio-libs/aiohttp
Author:
Author-email:
License: Apache 2
Location: c:\Users\devel\Documents\Palo Alto\dish-score-cards\.venv\Lib\site-packages
Requires: aiohappyeyeballs, aiosignal, attrs, frozenlist, multidict, yarl
Required-by:
```

### multidict Version

```console
$ python -m pip show multidict
Name: multidict
Version: 6.0.5
Summary: multidict implementation
Home-page: https://github.com/aio-libs/multidict
Author: Andrew Svetlov
Author-email: andrew.svetlov@gmail.com
License: Apache 2
Location: c:\Users\devel\Documents\Palo Alto\dish-score-cards\.venv\Lib\site-packages
Requires:
Required-by: aiohttp, yarl
```

### yarl Version

```console
$ python -m pip show yarl
Name: yarl
Version: 1.9.4
Summary: Yet another URL library
Home-page: https://github.com/aio-libs/yarl
Author: Andrew Svetlov
Author-email: andrew.svetlov@gmail.com
License: Apache-2.0
Location: c:\Users\devel\Documents\Palo Alto\dish-score-cards\.venv\Lib\site-packages
Requires: idna, multidict
Required-by: aiohttp
```

### OS

Windows 11

### Related component

Client

### Additional context

_No response_

### Code of Conduct

- [X] I agree to follow the aio-libs Code of Conduct

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.