cloudflare / cloudflare/cloudflared

🐛 net/http: HTTP/1.x transport connection broken: http: ContentLength=<N> with Body length 0

Open
#906 1 comment 1 reaction 0 assignees View on GitHub
Priority: Normal Type: Bug
Dominant language
Go
Stars
15.6k
Forks
1.4k
PR merge metrics
No merged PRs in 30d

Description

**Describe the bug**
We don't really know if this is related to cloudflared or something internally for us, but we have a few % of our requests failing because of this reason:

```
2023-03-02T13:03:54Z ERR Request failed error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: net/http: HTTP/1.x transport connection broken: http: ContentLength=105 with Body length 0" connIndex=1 dest=https:/// ip=198.41.192.77 type=http
```
The only common denominator when looking in our logs is that only requests that contain request body seems to show up with this error - not a single GET request (without body) has this error.

**To Reproduce**
We have tried to reproduce it using load tests using `wrk` by sending several 10's of thousands of requests (with req body) but haven't been able to.

cloudflared config:
```yaml
tunnel:
credentials-file: /etc/cloudflared/creds/credentials.json
metrics: 0.0.0.0:2000/ready
no-autoupdate: true
ingress:
- hostname:
service: http://.default.svc:3000
...
```

**Expected behavior**
Expect no errors with body requests. But we haven't been able to reproduce it so it's hard to know for sure if this is a problem in our code or not. It's just a hunch that this is a cloudflared problem at this point.

**Environment and versions**
- OS: Talos Linux on kubernetes. cloudflared runs on a Deployment with 2 replicas in the cluster.
- Architecture: AMD
- Version: cloudflare/cloudflared:2023.3.0

**Logs and errors**
See above

**Additional context**

We did find this stackoverflow thread. Could it be a bug in cloudflared? https://stackoverflow.com/questions/31337891/net-http-http-contentlength-222-with-body-length-0

We will continue investigate and will add more context/info if we find anything.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.