dragonflydb / dragonflydb/dragonfly

S3 snapshots does not work on non-AWS S3?

Open
#3,067 9 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
C++
Stars
31.6k
Forks
1.3k
Avg merge
1d 10h
Merged PRs (30d)
137

Description

**Describe the bug**
Dragonfly logs "InternalError" for everything S3.

Looking on the receiving end, my S3 service, it seems that Dragonfly makes HTTP requests to: `https://fqdn/https://fqdn/bucket/file` which of course does not work.

Log excerpts from startup and after a `BGSAVE`:

```
I20240521 12:20:35.599390 1 dfly_main.cc:646] Starting dragonfly df-v1.18.1-6851a4c845625b0b14bb145177322dafbbc9858e
(...)
I20240521 12:20:35.727556 9 snapshot_storage.cc:185] Creating AWS S3 client; region=us-east-1; https=true; endpoint=swift.elastx.cloud
I20240521 12:20:35.727806 9 credentials_provider_chain.cc:28] aws: disabled EC2 metadata
I20240521 12:20:35.730230 9 credentials_provider_chain.cc:36] aws: loaded credentials; provider=environment
I20240521 12:20:35.738584 10 snapshot_storage.cc:242] Load snapshot: Searching for snapshot in S3 path: s3://dragonfly-juicefs/
E20240521 12:21:01.488096 1 server_family.cc:816] Failed to load snapshot: Failed list objects in S3 bucket: InternalError
(...)
E20240521 14:06:23.187770 9 s3_write_file.cc:137] aws: s3 write file: failed to create multipart upload: InternalError
E20240521 14:06:48.932502 10 s3_write_file.cc:137] aws: s3 write file: failed to create multipart upload: InternalError
E20240521 14:06:48.940094 9 s3_write_file.cc:137] aws: s3 write file: failed to create multipart upload: InternalError
E20240521 14:06:48.940392 8 s3_write_file.cc:137] aws: s3 write file: failed to create multipart upload: InternalError
E20240521 14:06:48.944156 11 s3_write_file.cc:137] aws: s3 write file: failed to create multipart upload: InternalError
I20240521 14:06:48.944537 10 server_family.cc:1720] Error in BgSaveFb: Input/output error: Failed to open write file
```

**To Reproduce**
S3 credentials in environment variables `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`.

S3 endpoint in environment variable `DFLY_s3_endpoint` (in my case `swift.elastx.cloud`).

Snapshot dir set in environment variable `DFLY_dir` (in my case `s3://dragonfly-juicefs`).

**Expected behavior**
S3 snapshots to work.

**Environment (please complete the following information):**
- Kubernetes, Dragonfly Operator v1.1.2 with Dragonfly v1.18.1

**Additional context**
S3 API for Openstack Swift - not AWS

Contributor guide

Open the contributing guide

Research direction

Start with snapshot_storage.cc and s3_write_file.cc, then reproduce using DFLY_s3_endpoint, DFLY_dir, and the AWS credential environment variables against the stated Swift S3 service. Verify the generated request path and snapshot list, load, multipart upload, and BGSAVE behavior; done means non-AWS S3 endpoints no longer produce the duplicated URL or InternalError.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, cpp, kubernetes
Domain
backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.