googleapis / googleapis/google-cloud-python

Default expiration date is added to created table with a `TimePartitioning` when `expiration_ms` is explicitly none

Aberta
#15,741 14 comentários 0 reações 1 responsável Reivindicada por @chalmerlowe Ver no GitHub
api: bigquery external status: blocked type: feature request
Linguagem predominante
Python
Estrelas
5.4k
Forks
1.8k
Merge médio
2d 23h
PRs com merge (30d)
123

Descrição

#### Environment details
- OS type and version: Windows 10
- Python version: 3.9.0
- pip version: 21.1
- `google-cloud-bigquery` version: 2.13.1

#### Steps to reproduce / Code example

When running the following code a table is where the value of Partition Expiration is silently set to the default expiration date defined on the dataset.

```python
from google.cloud.bigquery import (
Table,
SchemaField,
TimePartitioning,
TimePartitioningType,
)

table = Table(
f"project.dataset.test_table",
schema=[
SchemaField("test", "STRING", "NULLABLE"),
SchemaField("date", "TIMESTAMP"),
]
)
table.time_partitioning = TimePartitioning(
TimePartitioningType.DAY, field="date", expiration_ms=None,
)

# bq being an already configured instance of google.cloud.bigquery.Client
bq.create_table(table)
```

To start it is good to note that the way to remove an expiration date (fully remove, not set to default) from a partition via the standard SQL is as follows:
```sql
ALTER TABLE ropo.orders
SET OPTIONS (partition_expiration_days=NULL)
```

So even though you explicitly seem to set the `expiration_ms` to "no expiration date" in the code it will still silently remove the table after time x (depending on the dataset). The field `expiration_ms` only accepts `None` and positive integers, therefore, when the dataset does have one by default, there is no way to create the table with a TimePartition but without an expiration_date at all. Values like `0`, `null`, and `NULL` will all throw an error.

I would advise changing the behavior as follows:

Set the default_values of some variable that is given the name `DEFAULT_PARTITION_EXPIRATION_FROM_DATASET` such that this behavior is clear from the function definition. Don't make the value of the variable `None` such that we still have this variable available to explicitly turn the expiration functionality off.

Guia de contribuição

Abrir o guia de contribuição

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.