difference between CUBEJS_DB_EXPORT_BUCKET, CUBESTORE_GCS_BUCKET and Local storage
- Dominant language
- Rust
- Stars
- 20.8k
- Forks
- 2.1k
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 181
Description
Hi I'm trying to understand a couple of things
My understanding is that with each cube-router/worker there are 2 levels of storage:
- remote storage
- local storage
Theres also another storage configuration:
- export buckets
- currently I have set `CUBEJS_DB_EXPORT_BUCKET` to a gcs bucket and I can see `.csv.gz` files within it.
The documentation points to the fact that remote storage should use a remote blob storage e.g. `gcs` to guarantee consistency.
My questions are
- what is the relationship between export buckets (storing .csv.gz files) and the remote storage (storing .parquet files)?
- why does cube store things in an export bucket then import it into a pod just to export it into another bucket again (if I understand the process correctly)?
- is remote storage accessed by all cubestore-worker/router nodes?
- what is local storage?
- do we need persisted volumes or can we just mount an empty volume each time a cubestore pod is created?
- does each worker/router pod need to point to the same local storage volume?
Contributor guide
Assessment
This issue has not been assessed yet.