apache / apache/superset

BigQuery: Copy SELECT / latest-partition WHERE uses PARSE_DATE('%Y%m%d', '__NULL__') and the query fails

Open
#44,155 1 comment 0 reactions 0 assignees View on GitHub
#bug
Dominant language
Python
Stars
74.8k
Forks
18.3k
Avg merge
2d 5h
Merged PRs (30d)
685

Description

### Bug description

When using a BigQuery time-partitioned table in SQL Lab, Copy SELECT statement to the clipboard (and table preview selectStar) automatically appends a latest-partition predicate. If the table has a __NULL__ partition (rows where the partition column is NULL), the generated SQL is invalid:

WHERE 'date' = PARSE_DATE('%Y%m%d', '__NULL__')

PARSE_DATE('%Y%m%d', '__NULL__') fails because __NULL__ is not a YYYYMMDD date. The same can happen with __UNPARTITIONED__.

**How to reproduce**
1. Connect a BigQuery database.
2. Open SQL Lab and select a table that is column-partitioned by a DATE field (e.g. date).
3. Ensure the table has at least one row where the partition column is NULL (BigQuery then has a __NULL__ partition in INFORMATION_SCHEMA.PARTITIONS).
4. In the table tree, click Copy SELECT statement to the clipboard.
5. Paste and run the SQL.

**Expected behavior**
The generated SELECT should filter on the latest real time partition (numeric YYYY / YYYYMM / YYYYMMDD / YYYYMMDDHH), not on BigQuery’s special partition IDs (__NULL__, __UNPARTITIONED__).

If there is no usable time partition, the extra WHERE should be omitted rather than emitting an invalid PARSE_DATE.

### Screenshots/recordings

### Superset version

6.1.0

### Python version

3.12

### Node version

18 or greater

### Browser

Chrome

### Additional context

Additional context
Database: Google BigQuery.
Table is time-unit column-partitioned (not ingestion-time). Partition field is a DATE column named date. The table contains NULL values in that column, so INFORMATION_SCHEMA.PARTITIONS includes partition_id = '__NULL__'.

This affects SQL Lab Copy SELECT statement and any other path that uses select_star(..., latest_partition=True) for BigQuery (table preview / sample).

### Checklist

- [x] I have searched Superset docs and Slack and didn't find a solution to my problem.
- [x] I have searched the GitHub issue tracker and didn't find a similar bug report.
- [ ] I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.

Contributor guide

Open the contributing guide

Research direction

Start at the BigQuery select_star(..., latest_partition=True) entry point used by SQL Lab Copy SELECT and table preview. Reproduce the case with __NULL__ and __UNPARTITIONED__ partition IDs, then verify that only a usable latest real partition produces a predicate and that no usable partition omits the extra WHERE.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, sql
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.