fluent / fluent/fluent-bit

bug: Runtime test flb-rt-out_cloudwatch "event_size_over_limit" hangs indefinitely (Deadlock in in_lib and flb_lib_push)

Open
#12,406 0 comments 0 reactions 0 assignees View on GitHub
status: waiting-for-triage
Dominant language
C
Stars
8.1k
Forks
2k
Avg merge
4d 16h
Merged PRs (30d)
58

Description

## Bug Report

**Describe the bug**
The runtime test flb-rt-out_cloudwatch hangs indefinitely during the execution of the event_size_over_limit sub-test (eventually resulting in a CTest runner timeout / exit code 124).

**To Reproduce**
1. Compile Fluent Bit with runtime tests enabled.
2. Run CTest on the CloudWatch Logs runtime test:
```
ctest -VV -R flb-rt-out_cloudwatch
```
3. Observe that the test gets stuck right after printing:
```
82: Test event_size_over_limit... [2026/09/11 09:03:44.668] [ info] [fluent bit] version=5.1.2, commit=, pid=2760
...
82: [2026/09/11 09:03:41.718] [error] [input:lib:lib.0] flb_log_event_decoder_get_last_result error : Wrong root type

```

**Expected behavior**
The test should process the payload, truncate it, send it to the mocked CloudWatch Logs output, and exit successfully without blocking.

**Screenshots**
```
[output:cloudwatch_logs:cloudwatch_logs.0] worker #0 started\n82: [2026/09/11 09:03:38.642] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log stream from-fluent-test in log group fluent\n82: [2026/09/11 09:03:38.642] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Created log stream from-fluent-test\n82: [2026/09/11 09:03:38.642] [error] [output:cloudwatch_logs:cloudwatch_logs.0] Recieved code 200 but response was invalid, x-amzn-RequestId header not found\n82: [2026/09/11 09:03:38.642] [error] [output:cloudwatch_logs:cloudwatch_logs.0] Failed to send log events\n82: [2026/09/11 09:03:38.642] [error] [output:cloudwatch_logs:cloudwatch_logs.0] Failed to send events\n82: [2026/09/11 09:03:38.642] [ warn] [engine] failed to flush chunk \'2750-1789117417.745878771.flb\', retry in 8 seconds: task_id=0, input=lib.0 > output=cloudwatch_logs.0 (out_id=0)\n82: [2026/09/11 09:03:39.746] [ warn] [engine] service will shutdown in max 5 seconds\n82: [2026/09/11 09:03:39.746] [ info] [engine] pausing all inputs..\n82: [2026/09/11 09:03:39.746] [error] [output:cloudwatch_logs:cloudwatch_logs.0] Recieved code 200 but response was invalid, x-amzn-RequestId header not found\n82: [2026/09/11 09:03:39.746] [error] [output:cloudwatch_logs:cloudwatch_logs.0] Failed to send log events\n82: [2026/09/11 09:03:39.746] [error] [output:cloudwatch_logs:cloudwatch_logs.0] Failed to send events\n82: [2026/09/11 09:03:39.746] [error] [engine] chunk \'2750-1789117417.745878771.flb\' cannot be retried: task_id=0, input=lib.0 > output=cloudwatch_logs.0\n82: [2026/09/11 09:03:40.642] [ info] [engine] service has stopped (0 pending tasks)\n82: [2026/09/11 09:03:40.642] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] thread worker #0 stopping...\n82: [2026/09/11 09:03:40.642] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] thread worker #0 stopped\n82: [ OK ]\n82: Test create_stream_escapes_json... [ OK ]\n82: Test put_events_escapes_stream_name... [ OK ]\n82: Test event_size_at_limit... [2026/09/11 09:03:40.678] [ info] [fluent bit] version=5.1.2, commit=, pid=2756\n82: [2026/09/11 09:03:40.678] [ info] [storage] ver=1.5.4, type=memory, sync=normal, checksum=off, max_chunks_up=128\n82: [2026/09/11 09:03:40.678] [ info] [simd ] SSE2\n82: [2026/09/11 09:03:40.678] [ info] [cmetrics] version=2.2.2\n82: [2026/09/11 09:03:40.678] [ info] [ctraces ] version=0.7.1\n82: [2026/09/11 09:03:40.678] [ info] [input:lib:lib.0] initializing\n82: [2026/09/11 09:03:40.678] [ info] [input:lib:lib.0] storage_strategy=\'memory\' (memory only)\n82: [2026/09/11 09:03:41.713] [ warn] [net] sync io_read #28 timeout after 1 seconds from: tcp://169.254.169.254:80\n82: [2026/09/11 09:03:41.713] [error] [http_client] broken connection to 169.254.169.254:80 ?\n82: [2026/09/11 09:03:41.713] [ warn] [imds] unable to evaluate IMDS version\n82: [2026/09/11 09:03:41.713] [ info] [sp] stream processor started\n82: [2026/09/11 09:03:41.713] [ info] [engine] Shutdown Grace Period=5, Shutdown Input Grace Period=2\n82: [2026/09/11 09:03:41.714] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] worker #0 started\n82: [2026/09/11 09:03:41.718] [error] [input:lib:lib.0] flb_log_event_decoder_get_last_result error : Wrong root type\n82: [2026/09/11 09:03:43.715] [ warn] [engine] service will shutdown in max 5 seconds\n82: [2026/09/11 09:03:43.715] [ info] [engine] pausing all inputs..\n82: [2026/09/11 09:03:44.642] [ info] [engine] service has stopped (0 pending tasks)\n82: [2026/09/11 09:03:44.642] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] thread worker #0 stopping...\n82: [2026/09/11 09:03:44.642] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] thread worker #0 stopped\n82: [ OK ]\n82: Test event_size_over_limit... [2026/09/11 09:03:44.668] [ info] [fluent bit] version=5.1.2, commit=, pid=2760\n82: [2026/09/11 09:03:44.668] [ info] [storage] ver=1.5.4, type=memory, sync=normal, checksum=off, max_chunks_up=128\n82: [2026/09/11 09:03:44.668] [ info] [simd ] SSE2\n82: [2026/09/11 09:03:44.668] [ info] [cmetrics] version=2.2.2\n82: [2026/09/11 09:03:44.668] [ info] [ctraces ] version=0.7.1\n82: [2026/09/11 09:03:44.668] [ info] [input:lib:lib.0] initializing\n82: [2026/09/11 09:03:44.668] [ info] [input:lib:lib.0] storage_strategy=\'memory\' (memory only)\n', errors: b''
```

**Your Environment**

* Version used: Fluent 5.1.2

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start by running `ctest -VV -R flb-rt-out_cloudwatch` and focus on the `event_size_over_limit` sub-test. Trace the `in_lib` and `flb_lib_push` paths named in the issue alongside the runtime test entry. Done means the oversized payload test truncates and sends its payload, exits without hanging, and the CTest run passes.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, c
Domain
backend, cloud, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.