googleapis / googleapis/google-cloud-python

Default expiration date is added to created table with a `TimePartitioning` when `expiration_ms` is explicitly none

Đang mở
#15,741 14 bình luận 0 reaction 1 người được giao Được @chalmerlowe nhận Xem trên GitHub
api: bigquery external status: blocked type: feature request
Ngôn ngữ chính
Python
Star
5.4k
Fork
1.8k
Merge trung bình
3 ngày 4 giờ
Pull request đã merge (30 ngày)
122

Mô tả

#### Environment details
- OS type and version: Windows 10
- Python version: 3.9.0
- pip version: 21.1
- `google-cloud-bigquery` version: 2.13.1

#### Steps to reproduce / Code example

When running the following code a table is where the value of Partition Expiration is silently set to the default expiration date defined on the dataset.

```python
from google.cloud.bigquery import (
Table,
SchemaField,
TimePartitioning,
TimePartitioningType,
)

table = Table(
f"project.dataset.test_table",
schema=[
SchemaField("test", "STRING", "NULLABLE"),
SchemaField("date", "TIMESTAMP"),
]
)
table.time_partitioning = TimePartitioning(
TimePartitioningType.DAY, field="date", expiration_ms=None,
)

# bq being an already configured instance of google.cloud.bigquery.Client
bq.create_table(table)
```

To start it is good to note that the way to remove an expiration date (fully remove, not set to default) from a partition via the standard SQL is as follows:
```sql
ALTER TABLE ropo.orders
SET OPTIONS (partition_expiration_days=NULL)
```

So even though you explicitly seem to set the `expiration_ms` to "no expiration date" in the code it will still silently remove the table after time x (depending on the dataset). The field `expiration_ms` only accepts `None` and positive integers, therefore, when the dataset does have one by default, there is no way to create the table with a TimePartition but without an expiration_date at all. Values like `0`, `null`, and `NULL` will all throw an error.

I would advise changing the behavior as follows:

Set the default_values of some variable that is given the name `DEFAULT_PARTITION_EXPIRATION_FROM_DATASET` such that this behavior is clear from the function definition. Don't make the value of the variable `None` such that we still have this variable available to explicitly turn the expiration functionality off.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.