adafruit / adafruit/Adafruit_CircuitPython_Requests
response.text() memory consumption is too high
- Dominant language
- Python
- Stars
- 49
- Forks
- 39
- PR merge metrics
- No merged PRs in 30d
Description
Getting the following exception:
`memory allocation failed, allocating 32763 bytes`
I have a bit more than 3X memory free at the time of making the call to response.text().
https://github.com/adafruit/Adafruit_CircuitPython_Requests/blob/c567e1d70e469c147e6cd38373aba2a8b797b7ba/adafruit_requests.py#L327
What I've found is that each pass through the `iter_content` loop is consuming 2X `chunk_size` (so 2X the content in total) and then at the `self.close()` another allocation happens for the full content size, where it throws the exception. Unclear to me if that's coming from `.close` or from the `.join` in `content`:
https://github.com/adafruit/Adafruit_CircuitPython_Requests/blob/c567e1d70e469c147e6cd38373aba2a8b797b7ba/adafruit_requests.py#L292
I've found that increasing the `chunk_size` to 64 gets me working at a cost of 2.5X `chunk_size` per pass through the `iter_content` loop (but half as many passes).
https://github.com/adafruit/Adafruit_CircuitPython_Requests/blob/c567e1d70e469c147e6cd38373aba2a8b797b7ba/adafruit_requests.py#L327
First observed on `Adafruit CircuitPython 8.2.10 on 2024-02-14; Raspberry Pi Pico W with rp2040`
Also reproduced on `Adafruit CircuitPython 9.0.2 on 2024-03-28; Raspberry Pi Pico W with rp2040`
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in adafruit_requests.py around the content/close code at line 292 and the iter_content loop at line 327. Reproduce the response.text() allocation failure on the reported Raspberry Pi Pico W and CircuitPython versions, then trace the repeated allocations. Done means the reported response can be read without the memory allocation exception.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100