firebase / firebase/extensions
bigquery-firestore-export: clearing PARTITIONING_FIELD is blocked but the DTS API allows it
- Dominant language
- TypeScript
- Stars
- 979
- Forks
- 433
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 125
Description
`bigquery-firestore-export` refuses to clear `PARTITIONING_FIELD`, but the Data Transfer API allows it.
Setting `PARTITIONING_FIELD` and later clearing it fails the update with:
> Cannot remove partitioning_field from an existing transfer config. The BigQuery Data Transfer API does not support clearing this parameter once it has been set. To change partitioning, you must create a new transfer config with the desired partitioning settings.
The API does support it. Creating a new transfer config is a costly workaround, since run history is stored under the old config ID and does not come with you.
### Reproduction
Run against a real project, `us` location:
1. Create a scheduled query with `partitioning_field: created_at` and destination template `probe_{run_time|"%H%M%S"}`. Two runs produced tables partitioned `DAY (field: created_at)`.
2. Clear it: `PATCH /v1/{name}?updateMask=params` with `"partitioning_field": ""`. Returns 200, and a follow-up `GET` reads it back as `""`.
3. Trigger another run. It carries `partitioning_field: ""`, reaches `SUCCEEDED`, and reports no error.
4. Compare the destination tables:
```
probe_152800 DAY (field: created_at) <- partitioning set
probe_152909 DAY (field: created_at) <- partitioning set
probe_153056 (none) <- after clearing
```
So the clear is accepted, persists, runs, and changes the output. It is not silently ignored and it does not fail.
The "you cannot un-partition an existing table" concern does not apply here: the destination template gives every run a new time-suffixed table, so nothing is mutated in place.
### Where it comes from
The guard and its comment are in `functions/src/dts.ts` in the upstream extension:
```js
// The BigQuery Data Transfer API does not support clearing this parameter
throw new Error(PARTITIONING_FIELD_REMOVAL_ERROR);
```
The kit inherited both in `a34ba233`. The kit and the extension differ only in what happens after the throw, so both are affected.
### Suggested fix
Drop the guard and let an empty `partitioning_field` through the update mask, and remove `PARTITIONING_FIELD_REMOVAL_ERROR` along with the comment asserting the limitation. Worth confirming against the extension too, since the belief originated there.
Not verified: behaviour in locations other than `us`, and whether a destination table without a time-suffixed template behaves differently.
Contributor guide
Research direction
Start in functions/src/dts.ts at the PARTITIONING_FIELD guard and its comment, then compare the corresponding upstream extension implementation inherited in commit a34ba233. Reproduce the PATCH against a real project in the us location and verify that clearing the field succeeds and changes a subsequent run; done means both implementations allow the empty value without the obsolete error constant.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- backend, data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100