Add missing tests for the `google.cloud.utils` BigQuery helpers
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
## Description
> This is an issue that is reserved for the Airflow Summit "Contributors" Workshop. This is denoted with the label `contributors-workshop`. Out of respect for the organizers and participants of this workshop, **please do not implement a PR that addresses this issue.**
>
> If this issue is still open following Airflow Summit, the label will be removed and the issue can be picked up.
Two modules in the `google` provider have no dedicated test module. Nothing under any `providers/*/tests/` directory imports either of them:
| Module | Expected test file |
| --- | --- |
| `providers/google/src/airflow/providers/google/cloud/utils/bigquery.py` | `providers/google/tests/unit/google/cloud/utils/test_bigquery.py` |
| `providers/google/src/airflow/providers/google/cloud/utils/bigquery_get_data.py` | `providers/google/tests/unit/google/cloud/utils/test_bigquery_get_data.py` |
Both are currently suppressed in the `OVERLOOKED_TESTS` allowlist in [`airflow-core/tests/unit/always/test_project_structure.py`](https://github.com/apache/airflow/blob/main/airflow-core/tests/unit/always/test_project_structure.py).
This is a scoped subset of the meta issue #35442, limited to the BigQuery helpers under `google/cloud/utils`.
### What should these tests cover?
Worth being precise about the current state, because it differs between the two modules and neither is quite "completely untested".
`bq_cast` and `convert_job_id` do get executed incidentally, through `BigQueryCursor` tests such as `test_next` in `providers/google/tests/unit/google/cloud/hooks/test_bigquery.py` and through `BigQueryInsertJobOperator` tests. But no test asserts their own semantics, and the interesting cases are all unexercised.
`bigquery_get_data` is worse than untested. The two places that call it, `bigquery_to_postgres.py` and `bigquery_to_sql.py`, are tested with the function patched out (`mock.patch("airflow.providers.google.cloud.transfers.bigquery_to_postgres.bigquery_get_data")`), so the real body never runs in CI at all.
For `bigquery.py`:
- `bq_cast` returns `None` for a `None` input regardless of the declared type
- `INTEGER` casts to `int`, and `FLOAT` casts to `float`
- `TIMESTAMP` also casts to `float`, which is deliberate but surprising enough that it should be pinned rather than left implicit
- `BOOLEAN` maps the strings `true` and `false` to real booleans, and anything else raises `ValueError` with the expected message
- An unrecognised type falls through and returns the string unchanged
- `convert_job_id` builds `project_id:location:job_id` for a single id, maps over the list form and returns a list, and defaults `location` to `US` when it is `None`
For `bigquery_get_data.py`:
- Successive batches are yielded as lists of row values, with `start_index` advancing by `batch_size` on each call to `BigQueryHook.list_rows`
- The generator terminates when a batch comes back empty, which is the only exit from the `itertools.count` loop, so a test that never returns an empty batch would hang
- `selected_fields` and `batch_size` are passed through to `list_rows` unmodified
- A `RowIterator` return raises `TypeError`, which is the guard against someone calling the hook with `return_iterator=True`
Two conventions from the contributing docs apply here. The hook should be mocked with `autospec` rather than a bare `Mock`, and since this module logs through an injected logger, any assertion on that should use structured `caplog` checks rather than matching raw log text.
## Definition of Done
1. Add the two test modules at the paths in the table above.
2. Remove the two corresponding entries from `OVERLOOKED_TESTS` in `airflow-core/tests/unit/always/test_project_structure.py`
3. Both of these tests should pass:
```bash
breeze testing providers-tests providers/google/tests/unit/google/cloud/utils/test_bigquery.py
breeze testing providers-tests providers/google/tests/unit/google/cloud/utils/test_bigquery_get_data.py
```
---
Drafted-by: Claude Code (Opus 5); reviewed and edited by @jroachgolf84 before posting
Contributor guide
Assessment
This issue has not been assessed yet.