apache / apache/arrow-adbc

go/adbc/driver/bigquery: list of potential enhancements, missing features and missing docs

Open
#1,974 3 comments 0 reactions 0 assignees View on GitHub
Type: enhancement
Dominant language
C#
Stars
627
Forks
217
Avg merge
22h 22m
Merged PRs (30d)
64

Description

### What feature or improvement would you like to see?

This todo list contains a number of potential enhancements, missing features and missing docs that I collected from the preliminary implementation of the BigQuery Go driver https://github.com/apache/arrow-adbc/pull/1722. Some of these may be implemented together in a single PR, while some others may need a bit more work to achieve.

Also, some of these features could be hard/impossible to implement with current BigQuery API or due to some other limitations in BigQuery.

### bigquery.QueryConfig

When doing a query, a `*bigquery.Query` is needed and it contains the query and the query options/configurations. Some of these configurable items (in `bigquery.QueryConfig`) are not implemented yet:

- [ ] TableDefinitions
- [ ] TimePartitioning
- [ ] RangePartitioning
- [ ] Clustering
- [ ] Labels
- [ ] DestinationEncryptionConfig
- [ ] SchemaUpdateOptions
- [ ] ConnectionProperties

*`Parameters`* *is implemented but I forgot to remove it from the mini todo list in the `statement.go`*.

### ADBC callbacks/features

- [ ] implement `ExecuteSchema`
- [ ] implement `ReadPartition`
- [ ] implement `ExecutePartitions`
- [ ] implement Substrait execution
- [ ] implement `GetInfo`, `GetTableSchema` and other functions for BigQuery's AdbcConnection and AdbcStatement
- [ ] implemented transactions
- [ ] implements `driverbase.DbObjectsEnumerator`
- [ ] GetObjectsCatalogs
- [ ] GetObjectsDbSchemas
- [ ] GetObjectsTables

### Missing Types
#### From BigQuery to Arrow

- [ ] `bigquery.IntervalFieldType`
- [ ] `bigquery.RangeFieldType`
- [ ] `bigquery.GeographyFieldType`, this one is returned as strings for now. But we can potentially consider using GeoArrow for this.

#### From Arrow to BigQuery

- [ ] `arrow.DURATION`, I'm not sure which SQL DataType would be a good representation for it. `DATETIME` could be a potential one for it if we count from `0001-01-01T00:00:00.000000Z`
- [ ] `arrow.INTERVAL_MONTHS`
- [ ] `arrow.INTERVAL_DAY_TIME`
- [ ] `arrow.INTERVAL_MONTH_DAY_NANO`, `DATETIME` could be a potential fit for all interval types, but the issue is there're no rules on how many days should be in a month.
- [ ] `arrow.RUN_END_ENCODED`
- [ ] `arrow.SPARSE_UNION`
- [ ] `arrow.DENSE_UNION`
- [ ] `arrow.DICTIONARY`
- [ ] `arrow.MAP`

### Docs

- [ ] Update the documentation, a list of supported options and their expected values. Related comment, https://github.com/apache/arrow-adbc/pull/1722#discussion_r1610279474

Contributor guide

Open the contributing guide

Research direction

Review the preliminary Go BigQuery driver implementation from PR 1722, starting with statement.go and the listed ADBC callbacks and BigQuery QueryConfig options. First scope one checklist item and verify its BigQuery API support; done means the selected capability is implemented with coverage and the supported options and expected values are documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, google-cloud
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.