fluent / fluent/fluent-bit

out_s3 to GCS: successful uploads (HTTP 200) judged as failed — `header_lookup()` matches `x-goog-stored-content-length` instead of `Content-Length` (explains #8525)

Open Beginner friendly
#12,267 0 comments 0 reactions 0 assignees View on GitHub
status: waiting-for-triage
Dominant language
C
Stars
8.1k
Forks
2k
Avg merge
4d 20h
Merged PRs (30d)
71

Description

# Bug Report

## Describe the bug

When using the `s3` output against Google Cloud Storage's S3-compatible XML API (`endpoint https://storage.googleapis.com`, HMAC auth, `use_put_object On`), **every** PutObject upload succeeds on the server side (GCS logs HTTP 200, object is created), but Fluent Bit judges **every** upload as failed and retries, producing duplicate objects indefinitely (one new object per attempt, since `$UUID` is regenerated per attempt). With default `net.io_timeout 0` the failure surfaces every ~4 minutes as `broken connection to storage.googleapis.com:443`.

I believe this is the root cause of #8525 ("s3 output for GCS repeat uploading same object", closed as not planned) and related to the misleading metrics described in #10440.

## Root cause

`header_lookup()` in `src/flb_http_client.c` searches response headers with an **unanchored, case-insensitive substring match**:

```c
/* Lookup the beginning of the header */
p = strcasestr(c->resp.data, header);
```

A GCS PutObject 200 response contains the header `x-goog-stored-content-length: ` **before** the real `Content-Length: 0`:

```
HTTP/1.1 200 OK
X-GUploader-UploadID: ...
ETag: "..."
x-goog-hash: crc32c=...
x-amz-checksum-crc32c: ...
x-goog-stored-content-length: 26 <-- tail of this name is "content-length: "
x-goog-stored-content-encoding: identity
Vary: Origin
Content-Length: 0 <-- the real one
Date: ...
Server: UploadServer
```

`strcasestr(resp, "Content-Length: ")` matches **inside** `x-goog-stored-content-length: 26` (the substring `content-length: ` at its tail), so `check_content_length()` sets `c->resp.content_length = 26` — the size of the object that was just uploaded — instead of `0`.

The client then waits for 26 bytes of response body that will never arrive:

- with `net.io_timeout 0` (default), it blocks until GCS resets the idle connection after ~4 minutes (confirmed with a packet capture: 200 response delivered and ACKed, ~4 min idle, then RST from the server side),
- `flb_http_do()` returns `-1` even though `c->resp.status` is already parsed as `200` (debug log shows `http_do=-1, HTTP Status: 200`),
- the AWS client / s3 output treats the attempt as failed and retries → a new object per attempt.

Since `x-goog-stored-content-length` is present on every GCS PutObject response and its value is always > 0, the failure is 100% deterministic. AWS S3 responses contain no header embedding the substring `content-length: `, which is why the same configuration works fine against AWS S3.

The unanchored match is still present on `master` (same code in `header_lookup()`).

## To Reproduce

1. Configure the s3 output against GCS with HMAC keys:

```
[OUTPUT]
Name s3
Match *
bucket my-bucket
region asia-northeast3
endpoint https://storage.googleapis.com
use_put_object On
total_file_size 1M
upload_timeout 1m
```

2. Send any log volume. Every upload logs:

```
[error] [http_client] broken connection to storage.googleapis.com:443 ?
[error] [output:s3:s3.0] PutObject request failed
```

(with a ~4 minute delay per attempt on default `net.io_timeout`), while the objects appear in the bucket successfully — one duplicate per retry.

3. Independent confirmation that the response itself is well-formed (real `Content-Length: 0` is present) — reproduce the response headers Fluent Bit receives:

```
curl -sv --http1.1 -X PUT --data-binary @test.txt \
--aws-sigv4 "aws:amz:asia-northeast3:s3" \
--user "$ACCESS_KEY:$SECRET_KEY" \
"https://storage.googleapis.com/my-bucket/test.txt" -o /dev/null
```

## Expected behavior

The `Content-Length` response header should be matched only at the beginning of a header line, `content_length` should be parsed as `0`, and the upload should be reported as successful after the 200 response.

## Suggested fix

Anchor the header search to line starts in `header_lookup()`:

```c
p = c->resp.data;
while ((p = strcasestr(p, header)) != NULL) {
/* only accept a match at the start of a line; otherwise a header
* name embedding another header name is matched by mistake,
* e.g. GCS's 'x-goog-stored-content-length' vs 'Content-Length' */
if (p == c->resp.data || *(p - 1) == '\n') {
break;
}
p++;
}
```

I'm happy to submit a PR with this change.

## Your Environment

- Version used: fluent/fluent-bit:5.0 (bug also present in current `master` source)
- Configuration: s3 output → GCS S3-compatible XML API, HMAC credentials, `use_put_object On`, TLS on
- Environment: GKE (containerd), DaemonSet tailing container logs
- Filters and plugins: tail → kubernetes → grep → rewrite_tag → s3

## Additional context

- Setting `net.io_timeout 30s` changes the failure cadence from ~4 min (server RST) to 30 s (client timeout) — consistent with the client waiting for body bytes that never arrive; it does not change the wrong judgment itself.
- Workaround we use in production: make retries idempotent by removing `$UUID` from `s3_key_format` (using a per-node unique suffix instead) together with `static_file_path On`, plus a low `retry_limit` — retries then overwrite the same object instead of accumulating duplicates.

Contributor guide

Open the contributing guide

Research direction

Start in src/flb_http_client.c at header_lookup() and inspect how response headers are located and how check_content_length() consumes the result. Reproduce the GCS response or use the provided configuration to verify that only the line-start Content-Length header is selected and that a successful 200 upload is no longer retried.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, google-cloud
Domain
cloud, networking
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.