GoogleCloudPlatform / GoogleCloudPlatform/datashare-toolkit
Optionally forestall duplicate ingestion
- Dominant language
- JavaScript
- Stars
- 94
- Forks
- 27
- PR merge metrics
- No merged PRs in 30d
Description
If the write disposition for an ingestion run is set to `WRITE_APPEND`, BigQuery will ingest the same file an arbitrary number of times, yielding duplicate records in the corresponding table which differ in their values for `bqds_batch_id` (which is shared by the entire ingestion iteration).
One possible way to avoid this duplication is to warn or fail if the same file is going to be uploaded twice by searching the existing batch ID for the same file name as the one incoming from GCS.
Matching on the file name could end up being a relatively unreliable approach depending on a publisher's specific naming conventions. An alternative to reduce false positives would be to use an MD5 hash or similar on the entire inbound file, and then use that for inbound file validation. While not foolproof, this should give both publisher namespace flexibility with static evaluation and rejection of precise duplicate inbound files.
Contributor guide
Assessment
This issue has not been assessed yet.