fluent / fluent/fluent-bit

[S3 OUTPUT] odd errors when workers are set

Open
#10,570 10 comments 4 reactions 0 assignees View on GitHub
status: waiting-for-triage
Dominant language
C
Stars
8.1k
Forks
2k
Avg merge
4d 20h
Merged PRs (30d)
71

Description

We're trying to get more throughput with our S3 output plugin as we noticed that, when using the rewrite_tag filter plugin (basically to split a single containers logs into X number of streams), that plugin is generating loads of _emitter files_ on disk and no amount of increasing the `upload_timeout` on the `S3` plugin seems to help. So we thought maybe using `workers` would work now as I know in the past the support wasn't there. We tried it and are getting odd results. Below are 2 different errors we see when setting `workers 2` that we don't see otherwise.

The emitter buffer directory seems to grow endlessly with thousands of very small files until our persistent volume runs out of space.

Error 1:
```
2025/07/09 22:08:05] [ info] [output:s3:s3.0] Successfully uploaded object /logs/application/cluster-integ-test/rate-limit-testing/web-container/hello-world/2025-07-09/stdout/220300-mETRinTB.gz
[2025/07/09 22:08:05] [ warn] [s3_key] Invalid Tag delimiter: does not exist in tag. tag=15613978119417515792-14826480349264227722, format=/logs/application/cluster-integ-test/$TAG[1]/$TAG[2]/$TAG[3]/%Y-%m-%d/$TAG[4]/%H%M%S-$UUID.gz
[2025/07/09 22:08:05] [ warn] [s3_key] Invalid / Out of bounds tag part: At most 10 tag parts ($TAG[0] - $TAG[9]) can be processed. tag=15613978119417515792-14826480349264227722, format=/logs/application/cluster-integ-test/$TAG[1]/$TAG[2]/$TAG[3]/%Y-%m-%d/$TAG[4]/%H%M%S-$UUID.gz, delimiters=.
[2025/07/09 22:08:05] [ info] [output:s3:s3.0] Successfully uploaded object /logs/application/cluster-integ-test/15613978119417515792-14826480349264227722[1]/15613978119417515792-14826480349264227722[2]/15613978119417515792-14826480349264227722[3]/2025-07-09/15613978119417515792-14826480349264227722[4]/220300-qOpanvDL.gz
[2025/07/09 22:08:05] [engine] caught signal (SIGSEGV)
#0 0x55af3d1b8d66 in cb_s3_flush() at plugins/out_s3/s3.c:2223
#1 0x55af3d6cd026 in co_init() at lib/monkey/deps/flb_libco/amd64.c:117
#2 0xffffffffffffffff in ???() at ???:0
```

Error 2:
```
[2025/07/09 22:09:07] [ info] [input:storage_backlog:storage_backlog.16] queueing emitter.17:1-1752097667.555759947.flb
[2025/07/09 22:09:07] [ info] [input:storage_backlog:storage_backlog.16] queueing emitter.17:1-1752097667.619441841.flb
#0 0x56005ab26fde in s3_store_file_delete() at plugins/out_s3/s3_store.c:424
#1 0x56005ab22a09 in upload_data() at plugins/out_s3/s3.c:1085
#2 0x56005ab24c86 in send_upload_request() at plugins/out_s3/s3.c:1662
#3 0x56005ab24dbf in s3_upload_queue() at plugins/out_s3/s3.c:1719
#4 0x56005ab25d66 in cb_s3_flush() at plugins/out_s3/s3.c:2223
#5 0x56005b03a026 in co_init() at lib/monkey/deps/flb_libco/amd64.c:117
#6 0xffffffffffffffff in ???() at ???:0
```

@edsiper @PettitWesley any ideas?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.