export bucket still leaves a pre-aggregation artifact in bigquery
- Dominant language
- Rust
- Stars
- 20.8k
- Forks
- 2.1k
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 181
Description
Hi, the following is my understanding please let me know if I'm misunderstanding anything:
- export buckets are enabled by setting `external: true` explicitly or by setting `EXTERNAL_DEFAULT` in env vars
- when using export bucket, cube will send the pre-aggregation to say, google cloud storage instead of its default functionality of saving the pre-aggregation in the source database i.e. bigquery in my case
If my understanding on the configurations are true, then for some reason I'm still seeing the same pre-aggregation being stored on bigquery even though export buckets are enabled. Why does it do this?
**envvar config**
```
cube_api/cube-refresh-worker:
- name: CUBEJS_DB_TYPE
value: "bigquery"
- name: CUBEJS_DB_BQ_PROJECT_ID
....
- name: CUBEJS_DB_BQ_CREDENTIALS
....
- CUBEJS_EXTERNAL_DEFAULT
value: "true"
- name: CUBEJS_DB_EXPORT_BUCKET
value: {GCS bucket name}
- name: CUBEJS_DB_EXPORT_BUCKET_TYPE
value: "gcp"
```
The docs arent super clear on what storing internally vs externally mean i.e. what setting `external: true` **reallly** does, I'm not sure I can find explicitly declared in the docs, similarly the docs for export buckets do not explain how it can be configured i.e. do you need to set the export bucket env vars AND also do some kind of special config or is just setting the export buckets environment variables sufficient? Furthermore, If my understanding is completely false and `external: true` has nothing to do with export buckets, then I think having a disclaimer somewhere on the documentation just to explicitly highlight the differences as well as documenting exactly what storing externally really means would be very helpful
Thanks!
Contributor guide
Assessment
This issue has not been assessed yet.