cube-js / cube-js/cube

Cubestore s3 fails when uploading with`File can't be listed after upload.` when subPath starts with /

Open
#6,136 1 comment 1 reaction 0 assignees View on GitHub
cube store help wanted
Dominant language
Rust
Stars
20.8k
Forks
2.1k
Avg merge
1d 2h
Merged PRs (30d)
181

Description

**Describe the bug**
Errors like:
```
Error during Metastore upload: CubeError { message: "File metastore-1675803621852-logs/2.flex can't be listed after upload. Either there's Cube Store cluster misconfiguration, or storage can't provide the required consistency.", backtrace: "", cause: Internal }
```
for every uploaded file to s3 if subpath starts with /

**To Reproduce**
Steps to reproduce the behavior:
Deploy cubestore with AWS s3 storage and use a subpath with leading `/`

**Expected behavior**
Error shouldnt happen as [AWS now provides strong read-after-write consistency](https://aws.amazon.com/about-aws/whats-new/2020/12/amazon-s3-now-delivers-strong-read-after-write-consistency-automatically-for-all-applications/)

**Screenshots**
Check [related slack thread](https://cube-js.slack.com/archives/C0206EXEVAS/p1668525573427129)

**Minimally reproducible Cube Schema**

**Version:**
master

**Additional context**
Related rust-s3 open issue: https://github.com/durch/rust-s3/issues/203

I fixed it by removing the leading `/` on my subpath, but cubestore should be able to manage this case silently.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.