out_s3 to GCS: successful uploads (HTTP 200) judged as failed — `header_lookup()` matches `x-goog-stored-content-length` instead of `Content-Length` (explains #8525)
- Dominant language
- C
- Stars
- 8.1k
- Forks
- 2k
- Avg merge
- 4d 20h
- Merged PRs (30d)
- 71
Description
# Bug Report
## Describe the bug
When using the `s3` output against Google Cloud Storage's S3-compatible XML API (`endpoint https://storage.googleapis.com`, HMAC auth, `use_put_object On`), **every** PutObject upload succeeds on the server side (GCS logs HTTP 200, object is created), but Fluent Bit judges **every** upload as failed and retries, producing duplicate objects indefinitely (one new object per attempt, since `$UUID` is regenerated per attempt). With default `net.io_timeout 0` the failure surfaces every ~4 minutes as `broken connection to storage.googleapis.com:443`.
I believe this is the root cause of #8525 ("s3 output for GCS repeat uploading same object", closed as not planned) and related to the misleading metrics described in #10440.
## Root cause
`header_lookup()` in `src/flb_http_client.c` searches response headers with an **unanchored, case-insensitive substring match**:
```c
/* Lookup the beginning of the header */
p = strcasestr(c->resp.data, header);
```
A GCS PutObject 200 response contains the header `x-goog-stored-content-length: ` **before** the real `Content-Length: 0`:
```
HTTP/1.1 200 OK
X-GUploader-UploadID: ...
ETag: "..."
x-goog-hash: crc32c=...
x-amz-checksum-crc32c: ...
x-goog-stored-content-length: 26 <-- tail of this name is "content-length: "
x-goog-stored-content-encoding: identity
Vary: Origin
Content-Length: 0 <-- the real one
Date: ...
Server: UploadServer
```
`strcasestr(resp, "Content-Length: ")` matches **inside** `x-goog-stored-content-length: 26` (the substring `content-length: ` at its tail), so `check_content_length()` sets `c->resp.content_length = 26` — the size of the object that was just uploaded — instead of `0`.
The client then waits for 26 bytes of response body that will never arrive:
- with `net.io_timeout 0` (default), it blocks until GCS resets the idle connection after ~4 minutes (confirmed with a packet capture: 200 response delivered and ACKed, ~4 min idle, then RST from the server side),
- `flb_http_do()` returns `-1` even though `c->resp.status` is already parsed as `200` (debug log shows `http_do=-1, HTTP Status: 200`),
- the AWS client / s3 output treats the attempt as failed and retries → a new object per attempt.
Since `x-goog-stored-content-length` is present on every GCS PutObject response and its value is always > 0, the failure is 100% deterministic. AWS S3 responses contain no header embedding the substring `content-length: `, which is why the same configuration works fine against AWS S3.
The unanchored match is still present on `master` (same code in `header_lookup()`).
## To Reproduce
1. Configure the s3 output against GCS with HMAC keys:
```
[OUTPUT]
Name s3
Match *
bucket my-bucket
region asia-northeast3
endpoint https://storage.googleapis.com
use_put_object On
total_file_size 1M
upload_timeout 1m
```
2. Send any log volume. Every upload logs:
```
[error] [http_client] broken connection to storage.googleapis.com:443 ?
[error] [output:s3:s3.0] PutObject request failed
```
(with a ~4 minute delay per attempt on default `net.io_timeout`), while the objects appear in the bucket successfully — one duplicate per retry.
3. Independent confirmation that the response itself is well-formed (real `Content-Length: 0` is present) — reproduce the response headers Fluent Bit receives:
```
curl -sv --http1.1 -X PUT --data-binary @test.txt \
--aws-sigv4 "aws:amz:asia-northeast3:s3" \
--user "$ACCESS_KEY:$SECRET_KEY" \
"https://storage.googleapis.com/my-bucket/test.txt" -o /dev/null
```
## Expected behavior
The `Content-Length` response header should be matched only at the beginning of a header line, `content_length` should be parsed as `0`, and the upload should be reported as successful after the 200 response.
## Suggested fix
Anchor the header search to line starts in `header_lookup()`:
```c
p = c->resp.data;
while ((p = strcasestr(p, header)) != NULL) {
/* only accept a match at the start of a line; otherwise a header
* name embedding another header name is matched by mistake,
* e.g. GCS's 'x-goog-stored-content-length' vs 'Content-Length' */
if (p == c->resp.data || *(p - 1) == '\n') {
break;
}
p++;
}
```
I'm happy to submit a PR with this change.
## Your Environment
- Version used: fluent/fluent-bit:5.0 (bug also present in current `master` source)
- Configuration: s3 output → GCS S3-compatible XML API, HMAC credentials, `use_put_object On`, TLS on
- Environment: GKE (containerd), DaemonSet tailing container logs
- Filters and plugins: tail → kubernetes → grep → rewrite_tag → s3
## Additional context
- Setting `net.io_timeout 30s` changes the failure cadence from ~4 min (server RST) to 30 s (client timeout) — consistent with the client waiting for body bytes that never arrive; it does not change the wrong judgment itself.
- Workaround we use in production: make retries idempotent by removing `$UUID` from `s3_key_format` (using a per-node unique suffix instead) together with `static_file_path On`, plus a low `retry_limit` — retries then overwrite the same object instead of accumulating duplicates.
Contributor guide
Research direction
Start in src/flb_http_client.c at header_lookup() and inspect how response headers are located and how check_content_length() consumes the result. Reproduce the GCS response or use the provided configuration to verify that only the line-start Content-Length header is selected and that a successful 200 upload is no longer retried.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, google-cloud
- Domain
- cloud, networking
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100