tailscale / tailscale/tailscale
Log streaming to S3 bucket - fails on S3-compatible "Backblaze B2"
- Dominant language
- Go
- Stars
- 36.5k
- Forks
- 3.2k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 123
Description
### What is the issue?
Backblaze B2 is an "S3 compatible" storage system. I was able to set it up as an endpoint for audit log streaming to an S3 bucket. The system authenticated correctly, then created a zero-length log file, and is now erroring with this:
> operation error S3: UploadPart, https response error StatusCode: 400, RequestID: d7257de11ed292b8, HostID: aM3AzYGEeNO834mUONMoz4zTEYdU3bTYC, api error InvalidArgument: Unsupported header 'x-amz-trailer' received for this API call.
The blog post at https://tailscale.com/blog/s3-log-streaming suggests that B2 did work at one time.
https://www.backblaze.com/docs/cloud-storage-use-the-aws-sdk-for-go-with-backblaze-b2 - the Backblaze documentation - refers to another set of unsupported headers. They recommend "Backblaze does not yet accept these headers, so we recommend downgrading to AWS Go v2 SDK version 1.72.3".
I opened a ticket on this but there might be more general interest in getting this solved, or recommending a specific self-hosted or otherwise compatible S3 interface.
### Steps to reproduce
1. Follow directions at https://tailscale.com/kb/1255/log-streaming
2. specifically, at https://tailscale.com/kb/1255/log-streaming?tab=s3-compatible
3. Enter credentials for my Backblaze B2 bucket
4. Observe that an empty file was created in that bucket
5. On the "logs" page, note the error "Streaming: failed"
6. Open ticket with support
7. File this Github issue TSS-59012
### Are there any recent changes that introduced the issue?
I'm sure that AWS changes S3 all the time, the better to make compatibility a moving target.
### OS
macOS
### OS version
macOS Sequoia 15.3.2
### Tailscale version
1.80.0 Mac App Store variant
### Other software
Backblaze B2
### Bug report
BUG-84b8ee8a91133af759ac96f0c193546301abf9dd6d09129b9ea1a0939c3fe68f-20250528232841Z-7833465d0f2cf0a9
Contributor guide
Assessment
This issue has not been assessed yet.