ESP8266WebServer memory exhaustion while processing requests
- Dominant language
- C++
- Stars
- 16.7k
- Forks
- 13.1k
- PR merge metrics
- No merged PRs in 30d
Description
### Basic Infos
- [X] This issue complies with the [issue POLICY doc](https://github.com/esp8266/Arduino/blob/master/POLICY.md).
- [X] I have read the documentation at [readthedocs](https://arduino-esp8266.readthedocs.io/en/latest) and the issue is not addressed there.
- [X] I have tested that the issue is present in current master branch (aka latest git).
- [X] I have searched the issue tracker for a similar issue.
- [X] I have filled out all fields below.
#### Platform
- Hardware: ESP-12
- Core Version: 5328a8b91ef04aa10cb9cb272ba1ade5299cb810
- Development Env: Arduino IDE
- Operating System: Windows
### Settings in IDE
- Module: Nodemcu
- Flash Size: 4MB/3MB
- lwip Variant: v2 Lower Memory
- CPU Frequency: 80Mhz
- Upload Using: SERIAL
- Upload Speed: 115200
### Problem Description
With the ESP8266WebServer, while processing requests, the String library is used heavily. The way that it is being used repeatedly duplicates data while processing it resulting in memory exhaustion. I'm trying to pass in some data to the web server via POST request, and would constantly get 0 args available. After debugging for a few hours, it turned out that there was no RAM left due to these String copies.
Example 1: ESP8266WebServer::urlDecode
This method reads in the source String character by character, and copies the data to a new String, appending to the end. Assuming there is nothing to decode (worst case scenario), this doubles the memory requirements for the particular argument at this stage of processing.
https://github.com/esp8266/Arduino/blob/5328a8b91ef04aa10cb9cb272ba1ade5299cb810/libraries/ESP8266WebServer/src/Parsing.cpp#L580
Example 2: ESP8266WebServer::_parseRequest
Pretty much the same as above, but on a larger scale. Rather than parsing URL parameters and POST data separately, they are concatenated together. This duplicates the entirety of POST data in RAM.
https://github.com/esp8266/Arduino/blob/5328a8b91ef04aa10cb9cb272ba1ade5299cb810/libraries/ESP8266WebServer/src/Parsing.cpp#L195
Example 3: substring
To separate out each argument, substring is used, duplicating each argument for processing.
https://github.com/esp8266/Arduino/blob/5328a8b91ef04aa10cb9cb272ba1ade5299cb810/libraries/ESP8266WebServer/src/Parsing.cpp#L325
There are countless more examples, too. All in all, passing in ~1KiB of data exhausted the ~14KiB of RAM free on the board while my application was running. ALL of these could be alleviated by doing inline processing on a single buffer, doing things C style rather than C++ style for the bulk of the internal processing. This would not only greatly reduce the overall memory footprint while processing the request, it would greatly speed it up too (less memory copying, less memory processing, and only 1 buffer that would need to be allocated/freed per request)
For my particular project, I've already started working on this conversion process. If there is desire, I can make this publicly available when I'm certain it is stable.
Also, I believe this could be the cause of countless other issues I've seen reported with the ESP8266WebServer. There are absolutely no errors/warnings whatsoever when this particular memory exhaustion happens, just the appearance of an empty request (or partiality request sometimes, depending on environment status).
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.