dns resolution for plugins using async mode does not consider all the DNS servers on the host
- Dominant language
- C
- Stars
- 8.1k
- Forks
- 2k
- Avg merge
- 4d 20h
- Merged PRs (30d)
- 71
Description
## Bug Report
**Describe the bug**
In fluent-bit, there can be plugins which are using [async mode for performance improvement](https://github.com/fluent/fluent-bit/blob/master/DEVELOPER_GUIDE.md#concurrency). This is the default setting and would be used by a lot of plugins.
Consider a scenario wherein the host has multiple DNS servers `[x.x.x.x, y.y.y.y]` such that the the first/primary DNS server `(x.x.x.x)` does not resolve the endpoint but the secondary DNS Server `(y.y.y.y)` resolves it correctly. In such cases, the plugins using async mode try the DNS resolution with the primary DNS only and fail without ever trying resolution with secondary DNS.
The error for the same is-
```
[ warn] [net] getaddrinfo(host='www.google.com', err=12): Timeout while contacting DNS servers
```
This is in contrast to how DNS resolution should happen. The expected behaviour is for resolution to be tried using all the servers in DNS Server list before conceding error.
**Note:** This scenario works perfectly for plugins wherein async mode is disabled.
**To Reproduce**
- Example log message if applicable:
```
[ warn] [net] getaddrinfo(host='www.google.com', err=12): Timeout while contacting DNS servers
```
- Steps to reproduce the problem:
The issue can be replicated by following the listed steps-
- Create a new Linux VM or just create a new container using `cr.fluentbit.io/fluent/fluent-bit:1.9.6-debug`
```
docker run -it cr.fluentbit.io/fluent/fluent-bit:1.9.4-debug bash
```
- Inside the container or VM, change the DNS setting to include an invalid DNS Server as the primary DNS server.
```
vi /etc/resolv.conf
# Change the file to include a new invalid nameserver. For example-
# search us-west-2.compute.internal
# nameserver 10.0.0.6 --> Invalid DNS Server
# nameserver 10.0.0.2 --> Valid DNS Server
```
- Start fluent-bit with http output plugin to send logs to a remote server.
```
export FLB_LOG_LEVEL=debug
./fluent-bit -i dummy -o http://www.google.com:443 -p tls=on -p tls.verify=off
```
Using `http://www.google.com:443` is the easiest repro of the issue as fluent-bit is unable to perform DNS resolution before any other thing happens. The same issue is applicable when using Kinesis Streams and Kinesis FIrehose plugins as well.
```
./fluent-bit -i dummy \
-o firehose \
-p "region=us-west-2" \
-p "delivery_stream=example-stream"
```
We could not use HTTP benchmarking server on localhost as we need to use DNS resolution for the same. The same can be set on another machine and used here to replicate the issue.
**Expected behavior**
DNS resolution should happen with the secondary DNS Server before erroring out.
For the first example (www.google.com), the logs cannot be sent since it is not a valid destination and therefore, we will get an error 405 from the Google server. This essentially means that the DNS resolution worked fine.
**Screenshots**
**Your Environment**
* Version used: 1.9.6
* Configuration: Dummy input plugin and any output plugin using http endpoint and async mode
* Environment name and version (e.g. Kubernetes? What version?): Docker
* Server type and version:
* Operating System and version: Linux and Windows OS
* Filters and plugins: Dummy input plugin and any output plugin using http endpoint and async mode
**Additional context**
Contributor guide
Assessment
This issue has not been assessed yet.