cloudflare / cloudflare/cloudflared
🐛 net/http: HTTP/1.x transport connection broken: http: ContentLength=<N> with Body length 0
- Dominant language
- Go
- Stars
- 15.6k
- Forks
- 1.4k
- PR merge metrics
- No merged PRs in 30d
Description
**Describe the bug**
We don't really know if this is related to cloudflared or something internally for us, but we have a few % of our requests failing because of this reason:
```
2023-03-02T13:03:54Z ERR Request failed error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: net/http: HTTP/1.x transport connection broken: http: ContentLength=105 with Body length 0" connIndex=1 dest=https:/// ip=198.41.192.77 type=http
```
The only common denominator when looking in our logs is that only requests that contain request body seems to show up with this error - not a single GET request (without body) has this error.
**To Reproduce**
We have tried to reproduce it using load tests using `wrk` by sending several 10's of thousands of requests (with req body) but haven't been able to.
cloudflared config:
```yaml
tunnel:
credentials-file: /etc/cloudflared/creds/credentials.json
metrics: 0.0.0.0:2000/ready
no-autoupdate: true
ingress:
- hostname:
service: http://.default.svc:3000
...
```
**Expected behavior**
Expect no errors with body requests. But we haven't been able to reproduce it so it's hard to know for sure if this is a problem in our code or not. It's just a hunch that this is a cloudflared problem at this point.
**Environment and versions**
- OS: Talos Linux on kubernetes. cloudflared runs on a Deployment with 2 replicas in the cluster.
- Architecture: AMD
- Version: cloudflare/cloudflared:2023.3.0
**Logs and errors**
See above
**Additional context**
We did find this stackoverflow thread. Could it be a bug in cloudflared? https://stackoverflow.com/questions/31337891/net-http-http-contentlength-222-with-body-length-0
We will continue investigate and will add more context/info if we find anything.
Contributor guide
Assessment
This issue has not been assessed yet.