cube-js / cube-js/cube

Cubestore Router space is growing continuously due to temp-uploads

Open
#9,114 14 comments 8 reactions 1 assignee Claimed by @igorlukanin View on GitHub
question
Dominant language
Rust
Stars
20.8k
Forks
2.1k
Avg merge
1d 2h
Merged PRs (30d)
181

Description

**Problem**

We're using self-hosted cubestore 1 router + 1 worker for preaggregations in production mode. There is also Cubejs API and refresh worker.
Space on router is growing due to temp-uploads directory which seems never purges. I see csv.gz files from day 0 and even files for preaggregations which don't exist anymore.

![Image](https://github.com/user-attachments/assets/30153885-fa18-4f04-97c8-6e38fa079c0d)

For storage we're using self-managed S3 compatible storage (similar to Minio) in the same k8s cluster as Cube.

Last versions tried - v1.1.0 and v1.1.15

Config or router is simple and uses standard env variables approach: CUBESTORE_SERVER_NAME, CUBESTORE_WORKERS, CUBESTORE_META_PORT, CUBESTORE_MINIO_SERVER_ENDPOINT, CUBESTORE_MINIO_BUCKET, CUBESTORE_MINIO_ACCESS_KEY_ID, CUBESTORE_MINIO_SECRET_ACCESS_KEY

My questions are:
- Any clue, why old and even non-existing preaggregations files are not deleted from temp-uploads on cubestore router? Maybe there is some config missed?
- How does cubestore local storage housekeeping works, could it be configured somehow or should be managed by some external cronjobs?
- Could it be an issue when from time too time we get in cubestore router logs something like `CubeError { message: "File sizes for cachestore-current doesn't match after upload. Expected to be 24 but 0 uploaded", backtrace: "", cause: Internal }` for metastore-* and cachestore-current? Files are present on S3 storage, it's just might be a short time lag after upload to make that file availabe due to metadata for files not added yet. Does it retry on checking file size in case of error or not?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.