Pass per-field BigQuery `OPTIONS` values to the LogicalPlan's Arrow Schema
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
I've been reading and learning the TableProvider APIs and have finally gotten around to taking a serious look at implementing support for `CREATE EXTERNAL TABLE`. I started down the road of using the statement level `OPTIONS` for adding field specific key/value config pairs similar to how the Parquet provider does with the `OPTION::COLUMN.NESTED.PATH` keys. However, my particular use case has quite a few field specific possible options that would be decently more ergonomic with field level options so I went looking for how I might be able to customize the SQL parser to support something of that nature.
After many false starts I ended up stumbling over the [BigQuery column options support](https://docs.rs/sqlparser/latest/sqlparser/ast/enum.ColumnOption.html#variant.Options) and thought that would be perfect. While I was a bit disappointed its not directly supported, it turned out to be relatively easy to plumb through so that's what I've done.
### Describe the solution you'd like
I'm not tied to anything specific beyond "allow for column specific key/value configuration options". The PR I'll open after filing this issue is one possible implementation, but it doesn't have to be *the* implementation by any means.
### Describe alternatives you've considered
My original thought was to basically that I was going to have to go the route used in the [`sql_dialect.rs`](https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/sql_dialect.rs) where I would basically have to reimplement the `CREATE EXTERNAL TABLE` logic and then somehow plumb my extra options around.
And the reason I went that direction was my first (hacky) though was to re-parse the `CreateExternalTable::definition` field but it apparently doesn't keep the actual original SQL around (at least via the cli AFAICT).
### Additional context
I've got no idea whether the reaction to this will be "Oh, neat!" or "Ewww, gross!". I'm not particularly tied to it because in the end I can always just go to the statement wide options, though given how easy it was to add it seems like it might be acceptable?
Contributor guide
Research direction
Start by reading the TableProvider APIs and the CREATE EXTERNAL TABLE path, then compare it with the custom handling in examples/sql_dialect.rs. Trace how the definition becomes a LogicalPlan and Arrow Schema, and determine where BigQuery column OPTIONS can be preserved. Done means field-specific key/value options reach the schema for provider use; no test file is named in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100