cube-js / cube-js/cube

CubeStore S3 transport failures leave abandoned multipart uploads without retry or cleanup

Open
#11,340 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
20.8k
Forks
2.1k
Avg merge
1d 2h
Merged PRs (30d)
181

Description

## Describe the bug

CubeStore can leave abandoned S3 multipart uploads when a cache-store checkpoint upload encounters a transport error or HTTP 503 response.

The most common errors observed are:

```text
Error during Cachestore upload: CubeError {
message: "AWS S3 error: reqwest: error sending request ...: channel closed",
cause: Internal
}
```
Other observed variants include:
```text
AWS S3 error: reqwest: error sending request ...: unexpected EOF
AWS S3 error: Got HTTP 503 with content ''
S3 delete returned non OK status: 503
File sizes for cachestore-current doesn't match after upload
```
A failed checkpoint leaves multipart uploads open. CubeStore creates another checkpoint later, but the MPUs belonging to the failed checkpoint remain in S3 until they are manually aborted or removed by an S3 lifecycle rule.

## Environment
CubeStore version: `1.6.69`
Deployment: Kubernetes on Amazon EKS
Remote storage: Amazon S3
Authentication: EKS IRSA / Web Identity
`CUBESTORE_MAX_ACTIVE_UPLOADS`: not explicitly set, so the default of `4` is used

## Observed behavior
During a six-hour production observation:
- 27 cache-store checkpoint upload failures occurred
- 23 were `channel closed`
- 1 was `unexpected EOF`
- 1 was an upload HTTP 503
- 1 was a delete HTTP 503
- 1 was a post-upload size verification failure
- 109 incomplete MPUs remained across 24 failed checkpoint directories

Newer checkpoints continued to complete and advance `cachestore-current`, confirming that the MPUs from the older checkpoint directories were stale.
The request volume appears far below the normal S3 per-prefix request-rate guidance, so this does not appear to be caused primarily by sustained S3 throttling.

## Relevant implementation
CubeStore calls `put_object_stream()` from its pinned rust-s3 dependency.
After initiating an MPU, transport errors are propagated here:
```rust
let response_data = request.response_data(true).await?;
```
Because the error is returned through `?`, `abort_upload()` is not called. The MPU is only aborted when a completed HTTP response has a non-2xx status:
```rust
if !(200..300).contains(&response_data.status_code()) {
match self.abort_upload(&path, upload_id).await {
// ...
}
}
```

The same issue appears possible during `CompleteMultipartUpload`, where a transport failure is also propagated without cleaning up the MPU:
```rust
let _response_data = complete_request.response_data(false).await?;
```
Relevant source:
https://github.com/cube-js/rust-s3/blob/c662b9c66c2929da185c46084fc5f455030ad75f/rust-s3/src/bucket.rs#L929-L1014

## Expected behavior
CubeStore should:
1. Retry transient transport errors and retryable HTTP responses such as 503 using exponential backoff.
2. Prefer a fresh connection for retries after connection-level failures.
3. Abort the MPU if uploading a part ultimately fails.
4. Abort the MPU if `CompleteMultipartUpload` ultimately fails.
5. Ensure cleanup occurs for every error path after an MPU has been initiated.
6. Log the S3 error code and response body when available, particularly for HTTP 503 responses.

## Suggested implementation
An MPU cleanup guard or equivalent `finally`-style mechanism could abort the upload unless it has been successfully completed.

- Retry handling should cover at least:
- HTTP 500, 502, 503, and 504
- connection closed/reset
- unexpected EOF
- timeout errors

Retries should use bounded exponential backoff with jitter.

Contributor guide

Open the contributing guide

Research direction

Start in rust-s3/src/bucket.rs around lines 929-1014, following put_object_stream and CompleteMultipartUpload response handling. Trace how transport errors and retryable HTTP responses propagate after an MPU begins, then verify that retries use bounded backoff and every ultimately failed upload is aborted while successful completions remain intact. Use the listed CubeStore checkpoint failure cases and S3 error details as the completion criteria.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, rust
Domain
backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.