BigQueryHook.create_table() crashes when table_resource is a Table, TableReference, or TableListItem instead of a dict
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
### Under which category would you file this issue?
Providers
### Apache Airflow version
main (unreleased)
### What happened and how to reproduce it?
`BigQueryHook.create_table()`'s `table_resource` parameter is typed as
`dict[str, Any] | Table | TableReference | TableListItem`, and the docstring explicitly documents
passing a bare reference ("If `table` is a reference, an empty table is created with the specified
ID"). But the implementation only actually works when `table_resource` is a plain `dict`:
```python
_table_resource: dict[str, Any] = {}
if isinstance(table_resource, Table):
_table_resource = Table.from_api_repr(table_resource) # type: ignore
if schema_fields:
_table_resource["schema"] = {"fields": schema_fields}
table_resource_final = {**table_resource, **_table_resource} # type: ignore
```
- Passing a real `Table` instance: `Table.from_api_repr(table_resource)` is backwards —
`from_api_repr` expects an API-JSON *dict*, not a `Table` object. Its first line does
`"tableReference" not in resource`, and since `Table` has no `__contains__`/`__iter__`, this
raises `TypeError: argument of type 'Table' is not iterable`.
- Passing a `TableReference` or `TableListItem`: neither is a `Table` subclass, so the `isinstance`
branch is skipped, and `{**table_resource, **_table_resource}` requires `table_resource` to be a
mapping. Neither class implements the mapping protocol, so this raises
`TypeError: 'TableReference' object is not a mapping` /
`'TableListItem' object is not a mapping`.
To reproduce:
```python
from airflow.providers.google.cloud.hooks.bigquery import BigQueryHook
hook = BigQueryHook()
table = hook.get_table(project_id="my-project", dataset_id="my_dataset", table_id="my_table")
hook.create_table(
dataset_id="my_dataset",
table_id="my_new_table",
table_resource=table, # a Table instance -> TypeError
)
```
This is reachable from a public path, not just internal to the hook —
`BigQueryCreateTableOperator` has the identical type hint on its own `table_resource` param and
forwards it straight into `bq_hook.create_table()` unmodified. A Dag author who follows the
documented type hint and passes a `Table`/`TableReference`/`TableListItem` — e.g. one obtained from
`BigQueryHook.get_table()` or `list_tables()`, which is the natural thing to do — hits a crash. All
existing unit tests for `create_table()` only ever pass a plain `dict`, so this path is untested.
Root cause: `Table`, `TableReference`, and `TableListItem` all expose `.to_api_repr()` to convert
themselves *into* a dict, but the code calls `Table.from_api_repr()` (dict *into* object) and does
raw `**`-unpacking on the object — both wrong direction / wrong assumption about the object's
protocol.
### What you think should happen instead?
`create_table()` should accept any of the types its own signature and docstring already promise
(`dict`, `Table`, `TableReference`, `TableListItem`) without raising, by converting each to a plain
dict via `.to_api_repr()` (wrapping a bare `TableReference` under `"tableReference"`, since unlike
`Table`/`TableListItem` its `to_api_repr()` is flat) before merging in `schema_fields` and building
the final `Table`.
### Operating System
_No response_
### Deployment
Other
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start in airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.create_table() and compare its table_resource handling with BigQueryCreateTableOperator's forwarding path. Review the existing create_table unit tests, then add coverage for Table, TableReference, and TableListItem inputs and verify schema_fields are merged into the resulting table without raising.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 74/100