fluent / fluent/fluent-bit

S3 output plugin getting SecondLevelDomainForbidden when used with Aliyun OSS

Open
#10,390 9 comments 0 reactions 0 assignees View on GitHub
Stale status: waiting-for-triage
Dominant language
C
Stars
8.1k
Forks
2k
Avg merge
4d 16h
Merged PRs (30d)
58

Description

## Bug Report

**Describe the bug**

When trying to upload logs to Aliyun OSS (Aliyun equivalent of S3) using the S3 output plugin, I get the following error:
```
PutObject API responded with error='SecondLevelDomainForbidden', message='Please use virtual hosted style to access.'
```

It seems that fluent-bit is using path-style S3 API, while OSS only supports virtual-host-style.
I couldn't find any config option to make fluent-bit use virtual-host-style S3 requests.

**To Reproduce**

- Example log message if applicable:
```
Fluent Bit v3.1.3
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

______ _ _ ______ _ _ _____ __
| ___| | | | | ___ (_) | |____ |/ |
| |_ | |_ _ ___ _ __ | |_ | |_/ /_| |_ __ __ / /`| |
| _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / \ \ | |
| | | | |_| | __/ | | | |_ | |_/ / | |_ \ V /.___/ /_| |_
\_| |_|\__,_|\___|_| |_|\__| \____/|_|\__| \_/ \____(_)___/

[2025/05/27 13:16:27] [ info] [fluent bit] version=3.1.3, commit=12a9de521c, pid=1
[2025/05/27 13:16:27] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2025/05/27 13:16:27] [ info] [cmetrics] version=0.9.1
[2025/05/27 13:16:27] [ info] [ctraces ] version=0.5.2
[2025/05/27 13:16:27] [ info] [input:cpu:cpu.0] initializing
[2025/05/27 13:16:27] [ info] [input:cpu:cpu.0] storage_strategy='memory' (memory only)
[2025/05/27 13:16:27] [ info] [fstore] created root path /tmp/fluent-bit/s3/REDACTED
[2025/05/27 13:16:27] [ info] [output:s3:s3.0] Using upload size 100000000 bytes
[2025/05/27 13:16:27] [ info] [sp] stream processor started
[2025/05/27 13:16:27] [ info] [output:s3:s3.0] worker #0 started
[2025/05/27 13:17:28] [error] [output:s3:s3.0] PutObject API responded with error='SecondLevelDomainForbidden', message='Please use virtual hosted style to access.'
[2025/05/27 13:17:28] [error] [output:s3:s3.0] Raw PutObject response: HTTP/1.1 403 Forbidden
Server: AliyunOSS
Date: Tue, 27 May 2025 13:17:28 GMT
Content-Type: application/xml
Content-Length: 376
Connection: keep-alive
x-amz-request-id: REDACTED
x-oss-ec: 0003-00001401

SecondLevelDomainForbidden
Please use virtual hosted style to access.
REDACTED
oss-cn-beijing-internal.aliyuncs.com
0003-00001401
https://api.aliyun.com/troubleshoot?q=0003-00001401

[2025/05/27 13:17:28] [error] [output:s3:s3.0] PutObject request failed
[2025/05/27 13:17:28] [error] [output:s3:s3.0] Could not send chunk with tag my_cpu

```

- Steps to reproduce the problem:

1. Create a bucket in Aliyun OSS
2. Set `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` env vars to the credentials of an Aliyun user who has access to that bucket
3. Run fluent-bit with the following config:

```
[SERVICE]
Flush 1
Daemon Off
Log_Level info
Parsers_File parsers.conf

[INPUT]
Name cpu
Tag my_cpu

[OUTPUT]
Name s3
Match my_cpu

region cn-beijing
endpoint https://oss-cn-beijing-internal.aliyuncs.com
bucket
s3_key_format /example-dir/example-file-%Y%m%d-%H%M%S_$UUID.log

upload_timeout 1m
use_put_object true
```

**Expected behavior**

Fluent-bit's S3 output plugin uses virtual-host-style S3 API when used with Aliyun OSS, either by automatically detecting it, or by providing a config option to switch to that type of API.

**Screenshots**

N/A

**Your Environment**

* Version used: 3.1.3 (docker image from `cr.fluentbit.io/fluent/fluent-bit:3.1.3-debug`)
* Configuration: as show in "Steps to reproduce"
* Environment name and version (e.g. Kubernetes? What version?): Kubernetes v1.30.7-aliyun.1
* Server type and version: N/A
* Operating System and version: N/A
* Filters and plugins: out_s3, some input plugin (happens with both `in_cpu` and `kubernetes_events`)

**Additional context**

We're trying to use fluent-bit in Kubernetes to upload various logs to S3-equivalent in various clouds we use.
In AWS it works fine, but in Aliyun we hit this error.

For now I found the following workaround:

- insert the bucket name to the endpint URL
- take the first segment of the S3 key and put it in the `bucket` parameter
- remove the first segment of the S3 key from the `s3_key_format` parameter

like this:
```
endpoint https://.oss-cn-beijing-internal.aliyuncs.com
bucket example-dir
s3_key_format /example-file-%Y%m%d-%H%M%S_$UUID.log
```

But this is an ugly hack and I'd rather not have to rely on it.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.