cube-js / cube-js/cube

difference between CUBEJS_DB_EXPORT_BUCKET, CUBESTORE_GCS_BUCKET and Local storage

Open
#6,436 3 comments 0 reactions 0 assignees View on GitHub
docs question
Dominant language
Rust
Stars
20.8k
Forks
2.1k
Avg merge
1d 2h
Merged PRs (30d)
181

Description

Hi I'm trying to understand a couple of things

My understanding is that with each cube-router/worker there are 2 levels of storage:
- remote storage
- local storage
Theres also another storage configuration:
- export buckets
- currently I have set `CUBEJS_DB_EXPORT_BUCKET` to a gcs bucket and I can see `.csv.gz` files within it.

The documentation points to the fact that remote storage should use a remote blob storage e.g. `gcs` to guarantee consistency.

My questions are
- what is the relationship between export buckets (storing .csv.gz files) and the remote storage (storing .parquet files)?
- why does cube store things in an export bucket then import it into a pod just to export it into another bucket again (if I understand the process correctly)?
- is remote storage accessed by all cubestore-worker/router nodes?
- what is local storage?
- do we need persisted volumes or can we just mount an empty volume each time a cubestore pod is created?
- does each worker/router pod need to point to the same local storage volume?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.