apache / apache/airflow

BigQueryHook.create_table() crashes when table_resource is a Table, TableReference, or TableListItem instead of a dict

Open
#73,091 1 comment 0 reactions 0 assignees View on GitHub
area:providers kind:bug provider:google
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Under which category would you file this issue?

Providers

### Apache Airflow version

main (unreleased)

### What happened and how to reproduce it?

`BigQueryHook.create_table()`'s `table_resource` parameter is typed as
`dict[str, Any] | Table | TableReference | TableListItem`, and the docstring explicitly documents
passing a bare reference ("If `table` is a reference, an empty table is created with the specified
ID"). But the implementation only actually works when `table_resource` is a plain `dict`:

```python
_table_resource: dict[str, Any] = {}
if isinstance(table_resource, Table):
_table_resource = Table.from_api_repr(table_resource) # type: ignore
if schema_fields:
_table_resource["schema"] = {"fields": schema_fields}
table_resource_final = {**table_resource, **_table_resource} # type: ignore
```

- Passing a real `Table` instance: `Table.from_api_repr(table_resource)` is backwards —
`from_api_repr` expects an API-JSON *dict*, not a `Table` object. Its first line does
`"tableReference" not in resource`, and since `Table` has no `__contains__`/`__iter__`, this
raises `TypeError: argument of type 'Table' is not iterable`.
- Passing a `TableReference` or `TableListItem`: neither is a `Table` subclass, so the `isinstance`
branch is skipped, and `{**table_resource, **_table_resource}` requires `table_resource` to be a
mapping. Neither class implements the mapping protocol, so this raises
`TypeError: 'TableReference' object is not a mapping` /
`'TableListItem' object is not a mapping`.

To reproduce:

```python
from airflow.providers.google.cloud.hooks.bigquery import BigQueryHook

hook = BigQueryHook()
table = hook.get_table(project_id="my-project", dataset_id="my_dataset", table_id="my_table")
hook.create_table(
dataset_id="my_dataset",
table_id="my_new_table",
table_resource=table, # a Table instance -> TypeError
)
```

This is reachable from a public path, not just internal to the hook —
`BigQueryCreateTableOperator` has the identical type hint on its own `table_resource` param and
forwards it straight into `bq_hook.create_table()` unmodified. A Dag author who follows the
documented type hint and passes a `Table`/`TableReference`/`TableListItem` — e.g. one obtained from
`BigQueryHook.get_table()` or `list_tables()`, which is the natural thing to do — hits a crash. All
existing unit tests for `create_table()` only ever pass a plain `dict`, so this path is untested.

Root cause: `Table`, `TableReference`, and `TableListItem` all expose `.to_api_repr()` to convert
themselves *into* a dict, but the code calls `Table.from_api_repr()` (dict *into* object) and does
raw `**`-unpacking on the object — both wrong direction / wrong assumption about the object's
protocol.

### What you think should happen instead?

`create_table()` should accept any of the types its own signature and docstring already promise
(`dict`, `Table`, `TableReference`, `TableListItem`) without raising, by converting each to a plain
dict via `.to_api_repr()` (wrapping a bare `TableReference` under `"tableReference"`, since unlike
`Table`/`TableListItem` its `to_api_repr()` is flat) before merging in `schema_fields` and building
the final `Table`.

### Operating System

_No response_

### Deployment

Other

### Deployment details

_No response_

### Anything else?

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start in airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.create_table() and compare its table_resource handling with BigQueryCreateTableOperator's forwarding path. Review the existing create_table unit tests, then add coverage for Table, TableReference, and TableListItem inputs and verify schema_fields are merged into the resulting table without raising.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.