Destination GCs: add hive partitioning in paths
- Ngôn ngữ chính
- Python
- Star
- 22.1k
- Fork
- 5.3k
- Chỉ số merge pull request
- Chỉ số pull request đang chờ
Mô tả
## Tell us about the problem you're trying to solve
**Edit: the GCS connector 'Bucket Path' setup has similar (but incomplete) functionality to the S3 connector but is undocumented. Critically the S3 connector includes an option to terminate the path with a `/` - overriding existing ${namespace} and ${stream} values in the path. This screenshot shows the error thrown when trying to perform the same functionality in GCS connector `(0.2.10)`: **

GCS connector has been really good for us so far, however, when trying to couple with federated queries in BigQuery, it can hurt your optimization because you are unable to partition the underlying table.
I'm hoping to show that Airbyte can support robust partitioning with some simple path manipulation on insert.
## Describe the solution you’d like
**Edit: we've confirmed in Airbyte slack much of this custom path functionality [exists in the S3 connector](https://docs.airbyte.com/integrations/destinations/s3/#step-2-set-up-the-s3-destination-connector-in-airbyte) and is available but undocumented in the GCS connector. **
I'd like to see a modal in the GCS destination configuration "Enable hive partitioning" with a field name value.
Hive partitioning is a GCS external table default option in BigQuery, and could really enhance a lot of lakehouse-type implementations. [Link to the hive partition spec.](https://cloud.google.com/bigquery/docs/hive-partitioned-queries-gcs)
**The current pathing as is:**
```
testing_bucket/data_output_path/public/users/2021_01_01_1609541171643_0.jsonl
↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑
| | | | | | | format extension
| | | | | | partition id
| | | | | upload time in millis
| | | | upload date in YYYY-MM-DD
| | | stream name
| | source namespace (if it exists)
| bucket path
bucket name
````
With a slight adjustment we would enable BigQuery to infer a partition (where ${field_name} is the partition field with inferred type `date`):
```
testing_bucket/data_output_path/public/users/${field_name}=2021-01-01/1609541171643_0.jsonl
↑ ↑
| upload date as partition
Chosen field name as partition field
```
## Describe the alternative you’ve considered or used
The alternative I've been doing to not read my entire bucket every time is to rapidly refresh the external table definition, dynamically creating paths to filter to the day. i.e. `testing_bucket/data_output_path/public/users/2021_01_01`
## Additional context
[External table definition link.
](https://cloud.google.com/bigquery/docs/external-table-definition)
[Hive partitioning spec link.](https://cloud.google.com/bigquery/docs/hive-partitioned-queries-gcs)
[Link to getOutputFilename in the GCS connector](https://github.com/airbytehq/airbyte/blob/d05755523007499c84028f6c37c80feefcf81bdf/airbyte-integrations/connectors/destination-gcs/src/main/java/io/airbyte/integrations/destination/gcs/writer/BaseGcsWriter.java#L141:~:text=//%20Filename%3A%20%3Cupload%2Ddate,%7D)
## Are you willing to submit a PR?
We do have a resource we can put on this, however as this is such a core connector I feel like our specific implementation may run into problems when trying to establish consensus.
Hướng dẫn đóng góp
Đánh giá
Issue này chưa được đánh giá.