dragonflydb / dragonflydb/dragonfly
S3 snapshots does not work on non-AWS S3?
- Dominant language
- C++
- Stars
- 31.6k
- Forks
- 1.3k
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 137
Description
**Describe the bug**
Dragonfly logs "InternalError" for everything S3.
Looking on the receiving end, my S3 service, it seems that Dragonfly makes HTTP requests to: `https://fqdn/https://fqdn/bucket/file` which of course does not work.
Log excerpts from startup and after a `BGSAVE`:
```
I20240521 12:20:35.599390 1 dfly_main.cc:646] Starting dragonfly df-v1.18.1-6851a4c845625b0b14bb145177322dafbbc9858e
(...)
I20240521 12:20:35.727556 9 snapshot_storage.cc:185] Creating AWS S3 client; region=us-east-1; https=true; endpoint=swift.elastx.cloud
I20240521 12:20:35.727806 9 credentials_provider_chain.cc:28] aws: disabled EC2 metadata
I20240521 12:20:35.730230 9 credentials_provider_chain.cc:36] aws: loaded credentials; provider=environment
I20240521 12:20:35.738584 10 snapshot_storage.cc:242] Load snapshot: Searching for snapshot in S3 path: s3://dragonfly-juicefs/
E20240521 12:21:01.488096 1 server_family.cc:816] Failed to load snapshot: Failed list objects in S3 bucket: InternalError
(...)
E20240521 14:06:23.187770 9 s3_write_file.cc:137] aws: s3 write file: failed to create multipart upload: InternalError
E20240521 14:06:48.932502 10 s3_write_file.cc:137] aws: s3 write file: failed to create multipart upload: InternalError
E20240521 14:06:48.940094 9 s3_write_file.cc:137] aws: s3 write file: failed to create multipart upload: InternalError
E20240521 14:06:48.940392 8 s3_write_file.cc:137] aws: s3 write file: failed to create multipart upload: InternalError
E20240521 14:06:48.944156 11 s3_write_file.cc:137] aws: s3 write file: failed to create multipart upload: InternalError
I20240521 14:06:48.944537 10 server_family.cc:1720] Error in BgSaveFb: Input/output error: Failed to open write file
```
**To Reproduce**
S3 credentials in environment variables `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`.
S3 endpoint in environment variable `DFLY_s3_endpoint` (in my case `swift.elastx.cloud`).
Snapshot dir set in environment variable `DFLY_dir` (in my case `s3://dragonfly-juicefs`).
**Expected behavior**
S3 snapshots to work.
**Environment (please complete the following information):**
- Kubernetes, Dragonfly Operator v1.1.2 with Dragonfly v1.18.1
**Additional context**
S3 API for Openstack Swift - not AWS
Contributor guide
Research direction
Start with snapshot_storage.cc and s3_write_file.cc, then reproduce using DFLY_s3_endpoint, DFLY_dir, and the AWS credential environment variables against the stated Swift S3 service. Verify the generated request path and snapshot list, load, multipart upload, and BGSAVE behavior; done means non-AWS S3 endpoints no longer produce the duplicated URL or InternalError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, cpp, kubernetes
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100