cube-js / cube-js/cube

How to configure retention of `*.csv.gz`, `*.parquet` and `*.chunk.parquet` files in GCP cloud storage?

Open
#6,653 1 comment 0 reactions 0 assignees View on GitHub
cube store question
Dominant language
Rust
Stars
20.8k
Forks
2.1k
Avg merge
1d 2h
Merged PRs (30d)
181

Description

**Problem**

How to cap the number of `*.csv.gz` files in cloud storage?

**Details**

We use the export bucket functionality with GCP cloud storage together with BigQuery. Cube instructs BigQuery to export buckets in to `*.csv.gz` whenever the refresh-scheduler recognizes a change. My question is how to handle retention of the `*.csv.gz` files?

- Are they needed long-term to restore caches of cubestore-workers, or are the `*.parquet` and `*.chunk.parquet` files used for this?
- If we can set a retention config, which one is recommended?
- Does cube manage retention?

Thank you!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.