adafruit / adafruit/Adafruit_CircuitPython_Requests

response.text() memory consumption is too high

Open
#183 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
49
Forks
39
PR merge metrics
No merged PRs in 30d

Description

Getting the following exception:

`memory allocation failed, allocating 32763 bytes`

I have a bit more than 3X memory free at the time of making the call to response.text().

https://github.com/adafruit/Adafruit_CircuitPython_Requests/blob/c567e1d70e469c147e6cd38373aba2a8b797b7ba/adafruit_requests.py#L327

What I've found is that each pass through the `iter_content` loop is consuming 2X `chunk_size` (so 2X the content in total) and then at the `self.close()` another allocation happens for the full content size, where it throws the exception. Unclear to me if that's coming from `.close` or from the `.join` in `content`:

https://github.com/adafruit/Adafruit_CircuitPython_Requests/blob/c567e1d70e469c147e6cd38373aba2a8b797b7ba/adafruit_requests.py#L292

I've found that increasing the `chunk_size` to 64 gets me working at a cost of 2.5X `chunk_size` per pass through the `iter_content` loop (but half as many passes).

https://github.com/adafruit/Adafruit_CircuitPython_Requests/blob/c567e1d70e469c147e6cd38373aba2a8b797b7ba/adafruit_requests.py#L327

First observed on `Adafruit CircuitPython 8.2.10 on 2024-02-14; Raspberry Pi Pico W with rp2040`

Also reproduced on `Adafruit CircuitPython 9.0.2 on 2024-03-28; Raspberry Pi Pico W with rp2040`

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in adafruit_requests.py around the content/close code at line 292 and the iter_content loop at line 327. Reproduce the response.text() allocation failure on the reported Raspberry Pi Pico W and CircuitPython versions, then trace the repeated allocations. Done means the reported response can be read without the memory allocation exception.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.