googleapis / googleapis/google-cloud-python

Default expiration date is added to created table with a `TimePartitioning` when `expiration_ms` is explicitly none

Open
#15,741 14 comments 0 reactions 1 assignee Claimed by @chalmerlowe View on GitHub
api: bigquery external status: blocked type: feature request
Dominant language
Python
Stars
5.4k
Forks
1.8k
Avg merge
3d 4h
Merged PRs (30d)
122

Description

#### Environment details
- OS type and version: Windows 10
- Python version: 3.9.0
- pip version: 21.1
- `google-cloud-bigquery` version: 2.13.1

#### Steps to reproduce / Code example

When running the following code a table is where the value of Partition Expiration is silently set to the default expiration date defined on the dataset.

```python
from google.cloud.bigquery import (
Table,
SchemaField,
TimePartitioning,
TimePartitioningType,
)

table = Table(
f"project.dataset.test_table",
schema=[
SchemaField("test", "STRING", "NULLABLE"),
SchemaField("date", "TIMESTAMP"),
]
)
table.time_partitioning = TimePartitioning(
TimePartitioningType.DAY, field="date", expiration_ms=None,
)

# bq being an already configured instance of google.cloud.bigquery.Client
bq.create_table(table)
```

To start it is good to note that the way to remove an expiration date (fully remove, not set to default) from a partition via the standard SQL is as follows:
```sql
ALTER TABLE ropo.orders
SET OPTIONS (partition_expiration_days=NULL)
```

So even though you explicitly seem to set the `expiration_ms` to "no expiration date" in the code it will still silently remove the table after time x (depending on the dataset). The field `expiration_ms` only accepts `None` and positive integers, therefore, when the dataset does have one by default, there is no way to create the table with a TimePartition but without an expiration_date at all. Values like `0`, `null`, and `NULL` will all throw an error.

I would advise changing the behavior as follows:

Set the default_values of some variable that is given the name `DEFAULT_PARTITION_EXPIRATION_FROM_DATASET` such that this behavior is clear from the function definition. Don't make the value of the variable `None` such that we still have this variable available to explicitly turn the expiration functionality off.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.