esp8266 / esp8266/Arduino

REGRESSION in 3.1.0: http download gets stuck after headers, as if not getting data

Open
#9,035 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
16.7k
Forks
13.1k
PR merge metrics
No merged PRs in 30d

Description

### Basic Infos

- [x] This issue complies with the [issue POLICY doc](https://github.com/esp8266/Arduino/blob/master/POLICY.md).
- [x] I have read the documentation at [readthedocs](https://arduino-esp8266.readthedocs.io/en/latest) and the issue is not addressed there.
- [x] I have tested that the issue is present in current master branch (aka latest git).
- [x] I have searched the issue tracker for a similar issue.
- [x] If there is a stack dump, I have decoded it.
- [x] I have filled out all fields below.

#### Platform

- Hardware: Sparkfun ESP8266 Thing Dev
- Core Version: I don't know, downloaded today
- Development Env: [Arduino IDE|]
- Operating System: [Manjaro Linux]

### Settings in IDE

- Module: [Sparkfun ESP8266 Thing Dev]
- Flash Mode: from the Arduino IDE
- Flash Size: no idea
- lwip Variant: no idea
- Reset Method: no idea
- Flash Frequency: [40Mhz]
- CPU Frequency: [80Mhz|160MHz]
- Upload Using: [SERIAL]
- Upload Speed: [115200] (serial upload only)

### Problem Description

In 2019 I wrote a sketch that downloads text from a url using **HTTPClient**. I uploaded it to a few boards and it has been working flawlessly, and still is.

Now I updated Arduino and the ESP8266 core to the latest version, re-compiled my code (I needed to change a call to `HTTPClient::begin()` because of BC-breaking changes in the API) and uploaded it to one of the boards), and now the download systematically gets stuck after downloading the http headers, waiting forever for the next bytes to become available.

The hardware is the same, the network is the same, the server I'm downloading from is the same, my code hasn't changed one bit (except again for the `HTTPClient::begin()` call). Literally the only thing that has changed is I switched to the latest version of Arduino IDE, the core and the libraries.

Code that used to work (that was well tested and has been working without issue for 4 years on a dozen of devices on different wifi networks) is now broken after the update.

Unfortunately I hadn't updated the libraries in the last 4 years, so I don't know when in the last 4 years the bug appeared.

I'm sorry I don't have the time to create a minimal but full working example, and I cannot share the full code because it's too complex. I'll share the part of the code that does the http download.

### [MCVE](https://stackoverflow.com/help/mcve) Sketch

```cpp

uint8_t httpbuf[128] = {0};

serial.printf("... Loading from url: %s\n", url.c_str());
int httpcode=200;
int len=KNOWN_LENGTH;
int bytesdownloaded=0;

http.begin(wificlient, url);
httpcode=http.GET();
if (httpcode==200) {
len=http.getSize();
}

serial.printf("... got HTTP code %d; size is %d\n", httpcode, len);
int ticks=0;
int ticks0bytes=0;

if (httpcode==304) {
serial.printf("No new file to download (HTTP 304 response)\n");
return 0;
}

if (httpcode==200) {
if (len==-1) {
serial.printf("Assuming size %d\n", KNOWN_LENGTH);
len = KNOWN_LENGTH;
}
ledCode(LEDCODE_HTTPGET_OK); // this is just a function that blinks the led with one of a set of given patterns

int timestarted = millis();
int timechecked = millis();
int timegotbytes = millis();

while (len > 0 || len ==-1) {
if (millis()-timestarted > DOWNLOAD_MAX_TIME*1000) {
serial.printf("Image download is taking too long. Aborting.\n");
return LIERR_TIMEOUT;
}
int gotbytes=0;

if (!http.connected()) {
break;
}
size_t avl=wificlient.available();
if (avl) {
gotbytes = wificlient.readBytes(httpbuf, ((avl>sizeof(httpbuf))?sizeof(httpbuf):avl));
bytesdownloaded+=gotbytes;
ticks0bytes=0;
}
else {
delay(100);
ticks0bytes++;
if (ticks0bytes>=20) {
serial.printf("... !! been waiting for available bytes for more than 2 seconds!!\n");
ticks0bytes=0;
ticks=10000;
}
}

if (gotbytes>0) {
if (len>0) len=max(0, len-gotbytes);
ticks+=gotbytes;
if (millis()-timechecked > 5000) {
timechecked=millis();
ticks+=10000;
}
if (ticks>=10000) {
ledCode(LEDCODE_HTTPSTREAM_GOTBYTES); // blink once
serial.printf("...((ticks = %d)) %d bytes downloaded; %d remaining\n", ticks, bytesdownloaded, len);
ticks=0;
}

for (int j=0; j DOWNLOAD_MAX_TIME_NODATA*1000) {
serial.printf("... !! No data for too long. Aborting.\n");
return LIERR_TIMEOUT;
}
}

}
serial.printf("Done downloading file (%d bytes downloaded).\n", bytesdownloaded);
ledCode(LEDCODE_IMAGE_DONE);

return 0;
}
else {
serial.printf("...HTTP Error %d\n", httpcode);
return LIERR_HTTP_ERROR;
}

```

### Debug Messages

The output from the code above is:

```
... Loading from url: ******************************************
... got HTTP code 200; size is 384000
... 0 bytes available.
... 0 bytes available.
... 0 bytes available.
... 0 bytes available.
... 0 bytes available.
... 0 bytes available.
... 0 bytes available.
... 0 bytes available.
... !! been waiting for available bytes for more than 2 seconds!!

[.... many times...]

... 0 bytes available.
... 0 bytes available.
... 0 bytes available.
... 0 bytes available.
... 0 bytes available.
... 0 bytes available.
... !! No data for too long. Aborting.
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.