fluent / fluent/fluentd

Fluentd in_http behind nginx, error 104

Open
#3,096 5 comments 1 reaction 0 assignees View on GitHub
bug
Dominant language
Ruby
Stars
13.6k
Forks
1.4k
Avg merge
1d 3h
Merged PRs (30d)
20

Description

**Describe the bug**
I was going to use configuration with fluentd and in_http plugin behind nginx (where nginx is just a proxy providing security). But faced with this issue in nginx logs while testing:
```
2020/08/04 13:27:05 [error] 436#436: *214 readv() failed (104: Connection reset by peer) while reading upstream, client: 192.168.122.1, server: localhost, request: "POST /test HTTP/1.1", upstream: "http://172.18.0.2:9880/test", host: "192.168.122.73"
```
It appears in around 15% cases of transmission. (15% of nginx log entries are the same as mentioned above). But despite this, all events pass through fluentd. I mean: they successfully leave it through output plugin. (tested on tens of events)
Here below is the simplest as possible configuration where the issue appears.

**To Reproduce**
Touch fluentd (behind nginx) with curl several times:
```
curl -X POST -d '{"foo":"bar"}' -H 'Content-Type: application/json' http://$nginx_ip/test
```
**Expected behavior**
Error entries should not be in nginx logs

**Your Environment**
- fluentd 1.11.1
- nginx/1.19.1
- Ubuntu 18.04.2 LTS
- 4.15.0-51-generic
- Docker version 19.03.6, build 369ce74a3c

**Your Configuration**
Fluentd official container: fluentd.conf
```

@type http

@type null

```
Nginx official container: just edit "location /" in default.conf and do "kill -HUP"
```
location / {
proxy_pass http://fluentd:9880;
}
```

**Your Error Log**
```
192.168.122.1 - - [04/Aug/2020:13:27:04 +0000] "POST /test HTTP/1.1" 200 0 "-" "curl/7.64.0" "-"
192.168.122.1 - - [04/Aug/2020:13:27:05 +0000] "POST /test HTTP/1.1" 200 0 "-" "curl/7.64.0" "-"
2020/08/04 13:27:05 [error] 436#436: *214 readv() failed (104: Connection reset by peer) while reading upstream, client: 192.168.122.1, server: localhost, request: "POST /test HTTP/1.1", upstream: "http://172.18.0.2:9880/test", host: "192.168.122.73"
192.168.122.1 - - [04/Aug/2020:13:27:06 +0000] "POST /test HTTP/1.1" 200 0 "-" "curl/7.64.0" "-"
192.168.122.1 - - [04/Aug/2020:13:27:07 +0000] "POST /test HTTP/1.1" 200 0 "-" "curl/7.64.0" "-"
192.168.122.1 - - [04/Aug/2020:13:27:08 +0000] "POST /test HTTP/1.1" 200 0 "-" "curl/7.64.0" "-"
192.168.122.1 - - [04/Aug/2020:13:27:08 +0000] "POST /test HTTP/1.1" 200 0 "-" "curl/7.64.0" "-"
192.168.122.1 - - [04/Aug/2020:13:27:09 +0000] "POST /test HTTP/1.1" 200 0 "-" "curl/7.64.0" "-"
2020/08/04 13:27:10 [error] 436#436: *226 readv() failed (104: Connection reset by peer) while reading upstream, client: 192.168.122.1, server: localhost, request: "POST /test HTTP/1.1", upstream: "http://172.18.0.2:9880/test", host: "192.168.122.73"
192.168.122.1 - - [04/Aug/2020:13:27:10 +0000] "POST /test HTTP/1.1" 200 0 "-" "curl/7.64.0" "-"
192.168.122.1 - - [04/Aug/2020:13:27:11 +0000] "POST /test HTTP/1.1" 200 0 "-" "curl/7.64.0" "-"
```

**Additional context**
It can be somehow related with #2963

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.