dbt-labs / dbt-labs/dbt-adapters
[Bug] BigQuery: Casting partition column disables partition pruning, causing expensive query execution
- Dominant language
- Python
- Stars
- 233
- Forks
- 362
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 9
Description
### Is this a new bug in dbt-core?
- [x] I believe this is a new bug in dbt-core
- [x] I have searched the existing issues, and I could not find an existing issue for this bug
### Current Behavior
**Description**:
When using dbt incremental or microbatching strategies on partitioned BigQuery tables, filters on the partition column (e.g. `timemsc`) are sometimes casted to timestamp. This disables partition pruning and results in very high query costs and execution time.
**Observed Behavior**:
* Query without cast (good): processed 137.82 MB
* Query with cast (bad): processed 736.15 GB
```sql
-- Good
where timemsc >= '2025-04-20' and timemsc < '2025-04-21'
-- Bad
where cast(timemsc as timestamp) >= '2025-04-20 00:00:00+00:00'
and cast(timemsc as timestamp) < '2025-04-21 00:00:00+00:00'
```
### Expected Behavior
The dbt framework should avoid applying casts to partition columns when filtering on date/datetime boundaries
### Steps To Reproduce
1. Use dbt with `incremental_strategy = 'insert_overwrite'` or `microbatching`
2. Model filters on a partitioned BigQuery table with a timestamp column
3. Observe query structure and bytes processed
### Relevant log output
```shell
```
### Environment
```markdown
* dbt version: Versionelless
* Adapter: dbt-bigquery
* Warehouse: BigQuery Standard SQL
```
### Which database adapter are you using with dbt?
_No response_
### Additional Context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.