How to configure retention of `*.csv.gz`, `*.parquet` and `*.chunk.parquet` files in GCP cloud storage?
Open
cube store
question
- Dominant language
- Rust
- Stars
- 20.8k
- Forks
- 2.1k
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 181
Description
**Problem**
How to cap the number of `*.csv.gz` files in cloud storage?
**Details**
We use the export bucket functionality with GCP cloud storage together with BigQuery. Cube instructs BigQuery to export buckets in to `*.csv.gz` whenever the refresh-scheduler recognizes a change. My question is how to handle retention of the `*.csv.gz` files?
- Are they needed long-term to restore caches of cubestore-workers, or are the `*.parquet` and `*.chunk.parquet` files used for this?
- If we can set a retention config, which one is recommended?
- Does cube manage retention?
Thank you!
Contributor guide
Assessment
This issue has not been assessed yet.