bug: Runtime test flb-rt-out_cloudwatch "event_size_over_limit" hangs indefinitely (Deadlock in in_lib and flb_lib_push)
- Dominant language
- C
- Stars
- 8.1k
- Forks
- 2k
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 58
Description
## Bug Report
**Describe the bug**
The runtime test flb-rt-out_cloudwatch hangs indefinitely during the execution of the event_size_over_limit sub-test (eventually resulting in a CTest runner timeout / exit code 124).
**To Reproduce**
1. Compile Fluent Bit with runtime tests enabled.
2. Run CTest on the CloudWatch Logs runtime test:
```
ctest -VV -R flb-rt-out_cloudwatch
```
3. Observe that the test gets stuck right after printing:
```
82: Test event_size_over_limit... [2026/09/11 09:03:44.668] [ info] [fluent bit] version=5.1.2, commit=, pid=2760
...
82: [2026/09/11 09:03:41.718] [error] [input:lib:lib.0] flb_log_event_decoder_get_last_result error : Wrong root type
```
**Expected behavior**
The test should process the payload, truncate it, send it to the mocked CloudWatch Logs output, and exit successfully without blocking.
**Screenshots**
```
[output:cloudwatch_logs:cloudwatch_logs.0] worker #0 started\n82: [2026/09/11 09:03:38.642] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log stream from-fluent-test in log group fluent\n82: [2026/09/11 09:03:38.642] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Created log stream from-fluent-test\n82: [2026/09/11 09:03:38.642] [error] [output:cloudwatch_logs:cloudwatch_logs.0] Recieved code 200 but response was invalid, x-amzn-RequestId header not found\n82: [2026/09/11 09:03:38.642] [error] [output:cloudwatch_logs:cloudwatch_logs.0] Failed to send log events\n82: [2026/09/11 09:03:38.642] [error] [output:cloudwatch_logs:cloudwatch_logs.0] Failed to send events\n82: [2026/09/11 09:03:38.642] [ warn] [engine] failed to flush chunk \'2750-1789117417.745878771.flb\', retry in 8 seconds: task_id=0, input=lib.0 > output=cloudwatch_logs.0 (out_id=0)\n82: [2026/09/11 09:03:39.746] [ warn] [engine] service will shutdown in max 5 seconds\n82: [2026/09/11 09:03:39.746] [ info] [engine] pausing all inputs..\n82: [2026/09/11 09:03:39.746] [error] [output:cloudwatch_logs:cloudwatch_logs.0] Recieved code 200 but response was invalid, x-amzn-RequestId header not found\n82: [2026/09/11 09:03:39.746] [error] [output:cloudwatch_logs:cloudwatch_logs.0] Failed to send log events\n82: [2026/09/11 09:03:39.746] [error] [output:cloudwatch_logs:cloudwatch_logs.0] Failed to send events\n82: [2026/09/11 09:03:39.746] [error] [engine] chunk \'2750-1789117417.745878771.flb\' cannot be retried: task_id=0, input=lib.0 > output=cloudwatch_logs.0\n82: [2026/09/11 09:03:40.642] [ info] [engine] service has stopped (0 pending tasks)\n82: [2026/09/11 09:03:40.642] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] thread worker #0 stopping...\n82: [2026/09/11 09:03:40.642] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] thread worker #0 stopped\n82: [ OK ]\n82: Test create_stream_escapes_json... [ OK ]\n82: Test put_events_escapes_stream_name... [ OK ]\n82: Test event_size_at_limit... [2026/09/11 09:03:40.678] [ info] [fluent bit] version=5.1.2, commit=, pid=2756\n82: [2026/09/11 09:03:40.678] [ info] [storage] ver=1.5.4, type=memory, sync=normal, checksum=off, max_chunks_up=128\n82: [2026/09/11 09:03:40.678] [ info] [simd ] SSE2\n82: [2026/09/11 09:03:40.678] [ info] [cmetrics] version=2.2.2\n82: [2026/09/11 09:03:40.678] [ info] [ctraces ] version=0.7.1\n82: [2026/09/11 09:03:40.678] [ info] [input:lib:lib.0] initializing\n82: [2026/09/11 09:03:40.678] [ info] [input:lib:lib.0] storage_strategy=\'memory\' (memory only)\n82: [2026/09/11 09:03:41.713] [ warn] [net] sync io_read #28 timeout after 1 seconds from: tcp://169.254.169.254:80\n82: [2026/09/11 09:03:41.713] [error] [http_client] broken connection to 169.254.169.254:80 ?\n82: [2026/09/11 09:03:41.713] [ warn] [imds] unable to evaluate IMDS version\n82: [2026/09/11 09:03:41.713] [ info] [sp] stream processor started\n82: [2026/09/11 09:03:41.713] [ info] [engine] Shutdown Grace Period=5, Shutdown Input Grace Period=2\n82: [2026/09/11 09:03:41.714] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] worker #0 started\n82: [2026/09/11 09:03:41.718] [error] [input:lib:lib.0] flb_log_event_decoder_get_last_result error : Wrong root type\n82: [2026/09/11 09:03:43.715] [ warn] [engine] service will shutdown in max 5 seconds\n82: [2026/09/11 09:03:43.715] [ info] [engine] pausing all inputs..\n82: [2026/09/11 09:03:44.642] [ info] [engine] service has stopped (0 pending tasks)\n82: [2026/09/11 09:03:44.642] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] thread worker #0 stopping...\n82: [2026/09/11 09:03:44.642] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] thread worker #0 stopped\n82: [ OK ]\n82: Test event_size_over_limit... [2026/09/11 09:03:44.668] [ info] [fluent bit] version=5.1.2, commit=, pid=2760\n82: [2026/09/11 09:03:44.668] [ info] [storage] ver=1.5.4, type=memory, sync=normal, checksum=off, max_chunks_up=128\n82: [2026/09/11 09:03:44.668] [ info] [simd ] SSE2\n82: [2026/09/11 09:03:44.668] [ info] [cmetrics] version=2.2.2\n82: [2026/09/11 09:03:44.668] [ info] [ctraces ] version=0.7.1\n82: [2026/09/11 09:03:44.668] [ info] [input:lib:lib.0] initializing\n82: [2026/09/11 09:03:44.668] [ info] [input:lib:lib.0] storage_strategy=\'memory\' (memory only)\n', errors: b''
```
**Your Environment**
* Version used: Fluent 5.1.2
**Additional context**
Contributor guide
Research direction
Start by running `ctest -VV -R flb-rt-out_cloudwatch` and focus on the `event_size_over_limit` sub-test. Trace the `in_lib` and `flb_lib_push` paths named in the issue alongside the runtime test entry. Done means the oversized payload test truncates and sends its payload, exits without hanging, and the CTest run passes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, c
- Domain
- backend, cloud, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100