out_forward plugin starts sending invalid packets
- Dominant language
- Ruby
- Stars
- 13.6k
- Forks
- 1.4k
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 20
Description
### Describe the bug
fluentd 1.18
I'm using the out_forward plugin to let it forward incoming logs to my proprietary log server. Every once in a while it starts sending invalid messages consisting of 7 bytes : 93 a5 7a 2e 6c 6f 67
Once fluentd enters this state is keeps sending this message over and over again, incoming logs get buffered in files until they reach their max size and new logs are no longer accepted and therefore lost.
I enabled trace level logging for the out_forward plugin
### To Reproduce
I cannot reproduce at will
### Expected behavior
fluentd out_forward plugin should not send invalid packets and if it does it should not do so over and over again
### Your Environment
```markdown
- Fluentd version: 1.18
- Package version:
- Operating system: RHEL 9 Linux
- Kernel version:
```
### Your Configuration
```apache
@type forward
@log_level trace
name
host 127.0.0.1
port 24225
ignore_network_errors_at_startup true
verify_connection_at_startup false
@type file
path //zlog.*.buf
flush_mode interval
flush_interval 1s
overflow_action throw_exception
total_limit_size 1GB
queued_chunks_limit_size 1024
compress gzip
retry_forever true
retry_type periodic
retry_wait 1s
```
### Your Error Log
```shell
2025-11-24 15:09:40 +0100 [trace]: dequeueing a chunk instance=2260
2025-11-24 15:09:40 +0100 [trace]: chunk dequeued instance=2260 metadata=#
2025-11-24 15:09:40 +0100 [trace]: trying flush for a chunk chunk="64457b6c5c3bd14110513c43a4d156cd"
2025-11-24 15:09:40 +0100 [trace]: adding write count instance=2240
2025-11-24 15:09:40 +0100 [trace]: executing sync write chunk="64457b6c5c3bd14110513c43a4d156cd"
2025-11-24 15:09:40 +0100 [debug]: connect new socket
2025-11-24 15:09:40 +0100 [trace]: write operation done, committing chunk="64457b6c5c3bd14110513c43a4d156cd"
2025-11-24 15:09:40 +0100 [trace]: committing write operation to a chunk chunk="64457b6c5c3bd14110513c43a4d156cd" delayed=false
2025-11-24 15:09:40 +0100 [trace]: purging a chunk instance=2260 chunk_id="64457b6c5c3bd14110513c43a4d156cd" metadata=#
2025-11-24 15:09:40 +0100 [trace]: chunk purged instance=2260 chunk_id="64457b6c5c3bd14110513c43a4d156cd" metadata=#
2025-11-24 15:09:40 +0100 [trace]: done to commit a chunk chunk="64457b6c5c3bd14110513c43a4d156cd"
2025-11-24 15:09:40.299441051 +0100 fluent.trace: {"instance":2260,"message":"dequeueing a chunk instance=2260"}
2025-11-24 15:09:40.299556774 +0100 fluent.trace: {"instance":2260,"metadata":"#","message":"chunk dequeued instance=2260 metadata=#"}
2025-11-24 15:09:40.299602102 +0100 fluent.trace: {"chunk":"64457b6c5c3bd14110513c43a4d156cd","message":"trying flush for a chunk chunk=\"64457b6c5c3bd14110513c43a4d156cd\""}
2025-11-24 15:09:40.299639559 +0100 fluent.trace: {"instance":2240,"message":"adding write count instance=2240"}
2025-11-24 15:09:40.299662757 +0100 fluent.trace: {"chunk":"64457b6c5c3bd14110513c43a4d156cd","message":"executing sync write chunk=\"64457b6c5c3bd14110513c43a4d156cd\""}
2025-11-24 15:09:40.299701023 +0100 fluent.debug: {"message":"connect new socket"}
2025-11-24 15:09:40.302766920 +0100 fluent.trace: {"chunk":"64457b6c5c3bd14110513c43a4d156cd","message":"write operation done, committing chunk=\"64457b6c5c3bd14110513c43a4d156cd\""}
2025-11-24 15:09:40.302835320 +0100 fluent.trace: {"chunk":"64457b6c5c3bd14110513c43a4d156cd","delayed":false,"message":"committing write operation to a chunk chunk=\"64457b6c5c3bd14110513c43a4d156cd\" delayed=false"}
2025-11-24 15:09:40.302873690 +0100 fluent.trace: {"instance":2260,"chunk_id":"64457b6c5c3bd14110513c43a4d156cd","metadata":"#","message":"purging a chunk instance=2260 chunk_id=\"64457b6c5c3bd14110513c43a4d156cd\" metadata=#"}
2025-11-24 15:09:40.302983926 +0100 fluent.trace: {"instance":2260,"chunk_id":"64457b6c5c3bd14110513c43a4d156cd","metadata":"#","message":"chunk purged instance=2260 chunk_id=\"64457b6c5c3bd14110513c43a4d156cd\" metadata=#"}
2025-11-24 15:09:40.303023355 +0100 fluent.trace: {"chunk":"64457b6c5c3bd14110513c43a4d156cd","message":"done to commit a chunk chunk=\"64457b6c5c3bd14110513c43a4d156cd\""}
2025-11-24 15:09:40 +0100 [trace]: writing events into buffer instance=2260 metadata_size=1
2025-11-24 15:09:40 +0100 [debug]: Created new chunk chunk_id="64457b6d67cb7d7c1d004aa1758ad9b9" metadata=#
2025-11-24 15:09:40 +0100 [trace]: chunk /ffdc/fluent/buffers/zlog.b64457b6d67cb7d7c1d004aa1758ad9b9.buf size_added: 159 new_size: 159
2025-11-24 15:09:40.617262760 +0100 fluent.trace: {"instance":2260,"metadata_size":1,"message":"writing events into buffer instance=2260 metadata_size=1"}
2025-11-24 15:09:40.617614515 +0100 fluent.debug: {"chunk_id":"64457b6d67cb7d7c1d004aa1758ad9b9","metadata":"#","message":"Created new chunk chunk_id=\"64457b6d67cb7d7c1d004aa1758ad9b9\" metadata=#"}
2025-11-24 15:09:40.617796147 +0100 fluent.trace: {"message":"chunk /ffdc/fluent/buffers/zlog.b64457b6d67cb7d7c1d004aa1758ad9b9.buf size_added: 159 new_size: 159"}
2025-11-24 15:09:40 +0100 [trace]: writing events into buffer instance=2260 metadata_size=1
2025-11-24 15:09:40 +0100 [trace]: chunk /ffdc/fluent/buffers/zlog.b64457b6d67cb7d7c1d004aa1758ad9b9.buf size_added: 159 new_size: 318
2025-11-24 15:09:40.618903549 +0100 fluent.trace: {"instance":2260,"metadata_size":1,"message":"writing events into buffer instance=2260 metadata_size=1"}
2025-11-24 15:09:40.619042316 +0100 fluent.trace: {"message":"chunk /ffdc/fluent/buffers/zlog.b64457b6d67cb7d7c1d004aa1758ad9b9.buf size_added: 159 new_size: 318"}
2025-11-24 15:09:40 +0100 [trace]: sending heartbeat host="127.0.0.1" port=24225 heartbeat_type=:transport
2025-11-24 15:09:40 +0100 [debug]: connect new socket
2025-11-24 15:09:40.983441674 +0100 fluent.trace: {"host":"127.0.0.1","port":24225,"heartbeat_type":"transport","message":"sending heartbeat host=\"127.0.0.1\" port=24225 heartbeat_type=:transport"}
2025-11-24 15:09:40.983567212 +0100 fluent.debug: {"message":"connect new socket"}
2025-11-24 15:09:41 +0100 [trace]: writing events into buffer instance=2260 metadata_size=1
2025-11-24 15:09:41 +0100 [trace]: chunk /ffdc/fluent/buffers/zlog.b64457b6d67cb7d7c1d004aa1758ad9b9.buf size_added: 159 new_size: 477
2025-11-24 15:09:41.122298570 +0100 fluent.trace: {"instance":2260,"metadata_size":1,"message":"writing events into buffer instance=2260 metadata_size=1"}
2025-11-24 15:09:41.122606090 +0100 fluent.trace: {"message":"chunk /ffdc/fluent/buffers/zlog.b64457b6d67cb7d7c1d004aa1758ad9b9.buf size_added: 159 new_size: 477"}
2025-11-24 15:09:41 +0100 [trace]: writing events into buffer instance=2260 metadata_size=1
2025-11-24 15:09:41 +0100 [trace]: chunk /ffdc/fluent/buffers/zlog.b64457b6d67cb7d7c1d004aa1758ad9b9.buf size_added: 159 new_size: 636
2025-11-24 15:09:41.129487541 +0100 fluent.trace: {"instance":2260,"metadata_size":1,"message":"writing events into buffer instance=2260 metadata_size=1"}
2025-11-24 15:09:41.129718161 +0100 fluent.trace: {"message":"chunk /ffdc/fluent/buffers/zlog.b64457b6d67cb7d7c1d004aa1758ad9b9.buf size_added: 159 new_size: 636"}
2025-11-24 15:09:41 +0100 [trace]: writing events into buffer instance=2260 metadata_size=1
2025-11-24 15:09:41.132771965 +0100 fluent.trace: {"instance":2260,"metadata_size":1,"message":"writing events into buffer instance=2260 metadata_size=1"}
2025-11-24 15:09:41 +0100 [trace]: chunk /ffdc/fluent/buffers/zlog.b64457b6d67cb7d7c1d004aa1758ad9b9.buf size_added: 7395 new_size: 8031
2025-11-24 15:09:41.133715266 +0100 fluent.trace: {"message":"chunk /ffdc/fluent/buffers/zlog.b64457b6d67cb7d7c1d004aa1758ad9b9.buf size_added: 7395 new_size: 8031"}
2025-11-24 15:09:41 +0100 [trace]: writing events into buffer instance=2260 metadata_size=1
2025-11-24 15:09:41.182099288 +0100 fluent.trace: {"instance":2260,"metadata_size":1,"message":"writing events into buffer instance=2260 metadata_size=1"}
2025-11-24 15:09:41 +0100 [trace]: chunk /ffdc/fluent/buffers/zlog.b64457b6d67cb7d7c1d004aa1758ad9b9.buf size_added: 159 new_size: 8190
2025-11-24 15:09:41.182447693 +0100 fluent.trace: {"message":"chunk /ffdc/fluent/buffers/zlog.b64457b6d67cb7d7c1d004aa1758ad9b9.buf size_added: 159 new_size: 8190"}
2025-11-24 15:09:41 +0100 [trace]: writing events into buffer instance=2260 metadata_size=1
2025-11-24 15:09:41.184271628 +0100 fluent.trace: {"instance":2260,"metadata_size":1,"message":"writing events into buffer instance=2260 metadata_size=1"}
2025-11-24 15:09:41 +0100 [trace]: chunk /ffdc/fluent/buffers/zlog.b64457b6d67cb7d7c1d004aa1758ad9b9.buf size_added: 159 new_size: 8349
2025-11-24 15:09:41.184573445 +0100 fluent.trace: {"message":"chunk /ffdc/fluent/buffers/zlog.b64457b6d67cb7d7c1d004aa1758ad9b9.buf size_added: 159 new_size: 8349"}
2025-11-24 15:09:41 +0100 [trace]: enqueueing all chunks in buffer instance=2260
2025-11-24 15:09:41 +0100 [trace]: enqueueing chunk instance=2260 metadata=#
2025-11-24 15:09:41.224905148 +0100 fluent.trace: {"instance":2260,"message":"enqueueing all chunks in buffer instance=2260"}
2025-11-24 15:09:41.225123287 +0100 fluent.trace: {"instance":2260,"metadata":"#","message":"enqueueing chunk instance=2260 metadata=#"}
```
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.