Log messages not retransmitted in case of network interruption

In a forwarding scenario with one graylog as messages collector and forwarder and a second graylog as central instance messages are not transmittet to the central graylog in case of network interruption.
Such kind of interruption can be a restart of a firewall located in between or a general internet issue.

Structure:

```
[local network]
|
[graylog forwarder] // collects messages from local network and forwards them to a central one
|
[Internet]
|
[firewall]
|
[central graylog] //receives message from forwarding instance
```

## Expected Behavior
1. Messages from the local network are continuously sent to the local graylog forwarder
2. graylog forwarder is configured with an Output to transmit those messages to the central one
3. In case of a network interruption, the forwarder / its output should recognize that the input of the forwarder is not available
4. During this time the forwarder is still receiving messages from the local network
5. The forwarder keeps track of those messages
6. When the central graylog and its input are available again, then the messages are transmitted.

Finally: all messages received by the forwarder and that should sent through a configured output should receive the central graylog instance.

## Current Behavior
Currently, when the input is not available, the output is still sending the next message in the queue to the input.

```
> Flags [P.], seq 8592478:8593108, ack 2541, win 274, options [nop,nop,TS val 78770560 ecr 2539788814], length 630
> Flags [P.], seq 8592478:8593108, ack 2541, win 274, options [nop,nop,TS val 78800640 ecr 2539788814], length 630
> Flags [P.], seq 8592478:8593108, ack 2541, win 274, options [nop,nop,TS val 78830720 ecr 2539788814], length 630
> Flags [P.], seq 8592478:8593108, ack 2541, win 274, options [nop,nop,TS val 78860800 ecr 2539788814], length 630
>
```

The TCP sequence numbers are the same.
Then after aprx. 15 minutes graylog detects the interrupted connection and initiates a new one.

`2018-08-31T06:45:42.189+02:00 INFO [GelfTcpTransport] Channel disconnected!`

But All Messages the forwarder received in between are not forwarded anymore.
(See Screenshots at: [https://community.graylog.org/t/log-messages-not-retransmitted-in-case-of-network-interruption/6668](url)

## Possible Solution
Meanwhile I tweaked the Linux TCP Settings on the forwarder, so that only some Messages may be lost by using:

```
/proc/sys/net/ipv4/tcp_retries1 to 1
/proc/sys/net/ipv4/tcp_retries2 to 1
```

Then it does not take 15 minutes to detect the interrupted connection. It is restarted after some seconds.

Anyway, tweaking some Linux kernel values should not be the solution here.

## Steps to Reproduce (for bugs)
Bug can be reproduced by:
1. building up the structure mentioned above. Maybe it works as well when two instances are directly connected.
2. configure an output and a corresponding input using GELF/TCP/TLS
3. continuously send messages to the forwarder, like 1 per second
4. stop the input on the central instance, wait 15 minutes and start it again.

## Context
I'm trying to establish a structure where a forwarder collects messages in a network and forward it to a central instance, but without any message loss.

## Your Environment

on forwarder and on central instance:

* Graylog Version: 2.4.6
* Elasticsearch Version: 5.6.11
* MongoDB Version: 3.6.7
* Operating System on central instance: Ubuntu 16.04.01, Kernel 4.15.0-1022-azure
* Operating System on forwarder: Ubuntu 16.04.05, Kernel 4.4.0-133-generic