googleapis / googleapis/google-cloud-python

Default expiration date is added to created table with a `TimePartitioning` when `expiration_ms` is explicitly none

Offen
#15,741 14 Kommentare 0 Reaktionen 1 zugewiesene Person Beansprucht von @chalmerlowe Auf GitHub ansehen
api: bigquery external status: blocked type: feature request
Vorherrschende Sprache
Python
Sterne
5.4k
Forks
1.8k
Ø Merge
3 T. 4 Std.
Gemergte PRs (30 T.)
122

Beschreibung

#### Environment details
- OS type and version: Windows 10
- Python version: 3.9.0
- pip version: 21.1
- `google-cloud-bigquery` version: 2.13.1

#### Steps to reproduce / Code example

When running the following code a table is where the value of Partition Expiration is silently set to the default expiration date defined on the dataset.

```python
from google.cloud.bigquery import (
Table,
SchemaField,
TimePartitioning,
TimePartitioningType,
)

table = Table(
f"project.dataset.test_table",
schema=[
SchemaField("test", "STRING", "NULLABLE"),
SchemaField("date", "TIMESTAMP"),
]
)
table.time_partitioning = TimePartitioning(
TimePartitioningType.DAY, field="date", expiration_ms=None,
)

# bq being an already configured instance of google.cloud.bigquery.Client
bq.create_table(table)
```

To start it is good to note that the way to remove an expiration date (fully remove, not set to default) from a partition via the standard SQL is as follows:
```sql
ALTER TABLE ropo.orders
SET OPTIONS (partition_expiration_days=NULL)
```

So even though you explicitly seem to set the `expiration_ms` to "no expiration date" in the code it will still silently remove the table after time x (depending on the dataset). The field `expiration_ms` only accepts `None` and positive integers, therefore, when the dataset does have one by default, there is no way to create the table with a TimePartition but without an expiration_date at all. Values like `0`, `null`, and `NULL` will all throw an error.

I would advise changing the behavior as follows:

Set the default_values of some variable that is given the name `DEFAULT_PARTITION_EXPIRATION_FROM_DATASET` such that this behavior is clear from the function definition. Don't make the value of the variable `None` such that we still have this variable available to explicitly turn the expiration functionality off.

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.