Cubestore Router space is growing continuously due to temp-uploads
- Dominant language
- Rust
- Stars
- 20.8k
- Forks
- 2.1k
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 181
Description
**Problem**
We're using self-hosted cubestore 1 router + 1 worker for preaggregations in production mode. There is also Cubejs API and refresh worker.
Space on router is growing due to temp-uploads directory which seems never purges. I see csv.gz files from day 0 and even files for preaggregations which don't exist anymore.

For storage we're using self-managed S3 compatible storage (similar to Minio) in the same k8s cluster as Cube.
Last versions tried - v1.1.0 and v1.1.15
Config or router is simple and uses standard env variables approach: CUBESTORE_SERVER_NAME, CUBESTORE_WORKERS, CUBESTORE_META_PORT, CUBESTORE_MINIO_SERVER_ENDPOINT, CUBESTORE_MINIO_BUCKET, CUBESTORE_MINIO_ACCESS_KEY_ID, CUBESTORE_MINIO_SECRET_ACCESS_KEY
My questions are:
- Any clue, why old and even non-existing preaggregations files are not deleted from temp-uploads on cubestore router? Maybe there is some config missed?
- How does cubestore local storage housekeeping works, could it be configured somehow or should be managed by some external cronjobs?
- Could it be an issue when from time too time we get in cubestore router logs something like `CubeError { message: "File sizes for cachestore-current doesn't match after upload. Expected to be 24 but 0 uploaded", backtrace: "", cause: Internal }` for metastore-* and cachestore-current? Files are present on S3 storage, it's just might be a short time lag after upload to make that file availabe due to metadata for files not added yet. Does it retry on checking file size in case of error or not?
Contributor guide
Assessment
This issue has not been assessed yet.