GoogleCloudPlatform / GoogleCloudPlatform/datashare-toolkit

Optionally forestall duplicate ingestion

Open
#83 4 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
JavaScript
Stars
94
Forks
27
PR merge metrics
No merged PRs in 30d

Description

If the write disposition for an ingestion run is set to `WRITE_APPEND`, BigQuery will ingest the same file an arbitrary number of times, yielding duplicate records in the corresponding table which differ in their values for `bqds_batch_id` (which is shared by the entire ingestion iteration).

One possible way to avoid this duplication is to warn or fail if the same file is going to be uploaded twice by searching the existing batch ID for the same file name as the one incoming from GCS.

Matching on the file name could end up being a relatively unreliable approach depending on a publisher's specific naming conventions. An alternative to reduce false positives would be to use an MD5 hash or similar on the entire inbound file, and then use that for inbound file validation. While not foolproof, this should give both publisher namespace flexibility with static evaluation and rejection of precise duplicate inbound files.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.