Fluentd in_http behind nginx, error 104
- Dominant language
- Ruby
- Stars
- 13.6k
- Forks
- 1.4k
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 20
Description
**Describe the bug**
I was going to use configuration with fluentd and in_http plugin behind nginx (where nginx is just a proxy providing security). But faced with this issue in nginx logs while testing:
```
2020/08/04 13:27:05 [error] 436#436: *214 readv() failed (104: Connection reset by peer) while reading upstream, client: 192.168.122.1, server: localhost, request: "POST /test HTTP/1.1", upstream: "http://172.18.0.2:9880/test", host: "192.168.122.73"
```
It appears in around 15% cases of transmission. (15% of nginx log entries are the same as mentioned above). But despite this, all events pass through fluentd. I mean: they successfully leave it through output plugin. (tested on tens of events)
Here below is the simplest as possible configuration where the issue appears.
**To Reproduce**
Touch fluentd (behind nginx) with curl several times:
```
curl -X POST -d '{"foo":"bar"}' -H 'Content-Type: application/json' http://$nginx_ip/test
```
**Expected behavior**
Error entries should not be in nginx logs
**Your Environment**
- fluentd 1.11.1
- nginx/1.19.1
- Ubuntu 18.04.2 LTS
- 4.15.0-51-generic
- Docker version 19.03.6, build 369ce74a3c
**Your Configuration**
Fluentd official container: fluentd.conf
```
@type http
@type null
```
Nginx official container: just edit "location /" in default.conf and do "kill -HUP"
```
location / {
proxy_pass http://fluentd:9880;
}
```
**Your Error Log**
```
192.168.122.1 - - [04/Aug/2020:13:27:04 +0000] "POST /test HTTP/1.1" 200 0 "-" "curl/7.64.0" "-"
192.168.122.1 - - [04/Aug/2020:13:27:05 +0000] "POST /test HTTP/1.1" 200 0 "-" "curl/7.64.0" "-"
2020/08/04 13:27:05 [error] 436#436: *214 readv() failed (104: Connection reset by peer) while reading upstream, client: 192.168.122.1, server: localhost, request: "POST /test HTTP/1.1", upstream: "http://172.18.0.2:9880/test", host: "192.168.122.73"
192.168.122.1 - - [04/Aug/2020:13:27:06 +0000] "POST /test HTTP/1.1" 200 0 "-" "curl/7.64.0" "-"
192.168.122.1 - - [04/Aug/2020:13:27:07 +0000] "POST /test HTTP/1.1" 200 0 "-" "curl/7.64.0" "-"
192.168.122.1 - - [04/Aug/2020:13:27:08 +0000] "POST /test HTTP/1.1" 200 0 "-" "curl/7.64.0" "-"
192.168.122.1 - - [04/Aug/2020:13:27:08 +0000] "POST /test HTTP/1.1" 200 0 "-" "curl/7.64.0" "-"
192.168.122.1 - - [04/Aug/2020:13:27:09 +0000] "POST /test HTTP/1.1" 200 0 "-" "curl/7.64.0" "-"
2020/08/04 13:27:10 [error] 436#436: *226 readv() failed (104: Connection reset by peer) while reading upstream, client: 192.168.122.1, server: localhost, request: "POST /test HTTP/1.1", upstream: "http://172.18.0.2:9880/test", host: "192.168.122.73"
192.168.122.1 - - [04/Aug/2020:13:27:10 +0000] "POST /test HTTP/1.1" 200 0 "-" "curl/7.64.0" "-"
192.168.122.1 - - [04/Aug/2020:13:27:11 +0000] "POST /test HTTP/1.1" 200 0 "-" "curl/7.64.0" "-"
```
**Additional context**
It can be somehow related with #2963
Contributor guide
Assessment
This issue has not been assessed yet.