tailscale / tailscale/tailscale

Log streaming to S3 bucket - fails on S3-compatible "Backblaze B2"

Open
#16,124 0 comments 1 reaction 0 assignees View on GitHub
bug
Dominant language
Go
Stars
36.5k
Forks
3.2k
Avg merge
2d 3h
Merged PRs (30d)
123

Description

### What is the issue?

Backblaze B2 is an "S3 compatible" storage system. I was able to set it up as an endpoint for audit log streaming to an S3 bucket. The system authenticated correctly, then created a zero-length log file, and is now erroring with this:

> operation error S3: UploadPart, https response error StatusCode: 400, RequestID: d7257de11ed292b8, HostID: aM3AzYGEeNO834mUONMoz4zTEYdU3bTYC, api error InvalidArgument: Unsupported header 'x-amz-trailer' received for this API call.

The blog post at https://tailscale.com/blog/s3-log-streaming suggests that B2 did work at one time.

https://www.backblaze.com/docs/cloud-storage-use-the-aws-sdk-for-go-with-backblaze-b2 - the Backblaze documentation - refers to another set of unsupported headers. They recommend "Backblaze does not yet accept these headers, so we recommend downgrading to AWS Go v2 SDK version 1.72.3".

I opened a ticket on this but there might be more general interest in getting this solved, or recommending a specific self-hosted or otherwise compatible S3 interface.

### Steps to reproduce

1. Follow directions at https://tailscale.com/kb/1255/log-streaming
2. specifically, at https://tailscale.com/kb/1255/log-streaming?tab=s3-compatible
3. Enter credentials for my Backblaze B2 bucket
4. Observe that an empty file was created in that bucket
5. On the "logs" page, note the error "Streaming: failed"
6. Open ticket with support
7. File this Github issue TSS-59012

### Are there any recent changes that introduced the issue?

I'm sure that AWS changes S3 all the time, the better to make compatibility a moving target.

### OS

macOS

### OS version

macOS Sequoia 15.3.2

### Tailscale version

1.80.0 Mac App Store variant

### Other software

Backblaze B2

### Bug report

BUG-84b8ee8a91133af759ac96f0c193546301abf9dd6d09129b9ea1a0939c3fe68f-20250528232841Z-7833465d0f2cf0a9

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.