cube-js / cube-js/cube

export bucket still leaves a pre-aggregation artifact in bigquery

Open
#6,772 0 comments 0 reactions 0 assignees View on GitHub
question
Dominant language
Rust
Stars
20.8k
Forks
2.1k
Avg merge
1d 2h
Merged PRs (30d)
181

Description

Hi, the following is my understanding please let me know if I'm misunderstanding anything:
- export buckets are enabled by setting `external: true` explicitly or by setting `EXTERNAL_DEFAULT` in env vars
- when using export bucket, cube will send the pre-aggregation to say, google cloud storage instead of its default functionality of saving the pre-aggregation in the source database i.e. bigquery in my case

If my understanding on the configurations are true, then for some reason I'm still seeing the same pre-aggregation being stored on bigquery even though export buckets are enabled. Why does it do this?

**envvar config**
```
cube_api/cube-refresh-worker:
- name: CUBEJS_DB_TYPE
value: "bigquery"
- name: CUBEJS_DB_BQ_PROJECT_ID
....
- name: CUBEJS_DB_BQ_CREDENTIALS
....
- CUBEJS_EXTERNAL_DEFAULT
value: "true"
- name: CUBEJS_DB_EXPORT_BUCKET
value: {GCS bucket name}
- name: CUBEJS_DB_EXPORT_BUCKET_TYPE
value: "gcp"
```

The docs arent super clear on what storing internally vs externally mean i.e. what setting `external: true` **reallly** does, I'm not sure I can find explicitly declared in the docs, similarly the docs for export buckets do not explain how it can be configured i.e. do you need to set the export bucket env vars AND also do some kind of special config or is just setting the export buckets environment variables sufficient? Furthermore, If my understanding is completely false and `external: true` has nothing to do with export buckets, then I think having a disclaimer somewhere on the documentation just to explicitly highlight the differences as well as documenting exactly what storing externally really means would be very helpful

Thanks!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.