googleapis / googleapis/google-cloud-python
Default expiration date is added to created table with a `TimePartitioning` when `expiration_ms` is explicitly none
- Dominant language
- Python
- Stars
- 5.4k
- Forks
- 1.8k
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 122
Description
#### Environment details
- OS type and version: Windows 10
- Python version: 3.9.0
- pip version: 21.1
- `google-cloud-bigquery` version: 2.13.1
#### Steps to reproduce / Code example
When running the following code a table is where the value of Partition Expiration is silently set to the default expiration date defined on the dataset.
```python
from google.cloud.bigquery import (
Table,
SchemaField,
TimePartitioning,
TimePartitioningType,
)
table = Table(
f"project.dataset.test_table",
schema=[
SchemaField("test", "STRING", "NULLABLE"),
SchemaField("date", "TIMESTAMP"),
]
)
table.time_partitioning = TimePartitioning(
TimePartitioningType.DAY, field="date", expiration_ms=None,
)
# bq being an already configured instance of google.cloud.bigquery.Client
bq.create_table(table)
```
To start it is good to note that the way to remove an expiration date (fully remove, not set to default) from a partition via the standard SQL is as follows:
```sql
ALTER TABLE ropo.orders
SET OPTIONS (partition_expiration_days=NULL)
```
So even though you explicitly seem to set the `expiration_ms` to "no expiration date" in the code it will still silently remove the table after time x (depending on the dataset). The field `expiration_ms` only accepts `None` and positive integers, therefore, when the dataset does have one by default, there is no way to create the table with a TimePartition but without an expiration_date at all. Values like `0`, `null`, and `NULL` will all throw an error.
I would advise changing the behavior as follows:
Set the default_values of some variable that is given the name `DEFAULT_PARTITION_EXPIRATION_FROM_DATASET` such that this behavior is clear from the function definition. Don't make the value of the variable `None` such that we still have this variable available to explicitly turn the expiration functionality off.
Contributor guide
Assessment
This issue has not been assessed yet.