firebase / firebase/extensions

bigquery-firestore-export: clearing PARTITIONING_FIELD is blocked but the DTS API allows it

Open
#2,985 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
979
Forks
433
Avg merge
1d 13h
Merged PRs (30d)
125

Description

`bigquery-firestore-export` refuses to clear `PARTITIONING_FIELD`, but the Data Transfer API allows it.

Setting `PARTITIONING_FIELD` and later clearing it fails the update with:

> Cannot remove partitioning_field from an existing transfer config. The BigQuery Data Transfer API does not support clearing this parameter once it has been set. To change partitioning, you must create a new transfer config with the desired partitioning settings.

The API does support it. Creating a new transfer config is a costly workaround, since run history is stored under the old config ID and does not come with you.

### Reproduction

Run against a real project, `us` location:

1. Create a scheduled query with `partitioning_field: created_at` and destination template `probe_{run_time|"%H%M%S"}`. Two runs produced tables partitioned `DAY (field: created_at)`.
2. Clear it: `PATCH /v1/{name}?updateMask=params` with `"partitioning_field": ""`. Returns 200, and a follow-up `GET` reads it back as `""`.
3. Trigger another run. It carries `partitioning_field: ""`, reaches `SUCCEEDED`, and reports no error.
4. Compare the destination tables:

```
probe_152800 DAY (field: created_at) <- partitioning set
probe_152909 DAY (field: created_at) <- partitioning set
probe_153056 (none) <- after clearing
```

So the clear is accepted, persists, runs, and changes the output. It is not silently ignored and it does not fail.

The "you cannot un-partition an existing table" concern does not apply here: the destination template gives every run a new time-suffixed table, so nothing is mutated in place.

### Where it comes from

The guard and its comment are in `functions/src/dts.ts` in the upstream extension:

```js
// The BigQuery Data Transfer API does not support clearing this parameter
throw new Error(PARTITIONING_FIELD_REMOVAL_ERROR);
```

The kit inherited both in `a34ba233`. The kit and the extension differ only in what happens after the throw, so both are affected.

### Suggested fix

Drop the guard and let an empty `partitioning_field` through the update mask, and remove `PARTITIONING_FIELD_REMOVAL_ERROR` along with the comment asserting the limitation. Worth confirming against the extension too, since the belief originated there.

Not verified: behaviour in locations other than `us`, and whether a destination table without a time-suffixed template behaves differently.

Contributor guide

Open the contributing guide

Research direction

Start in functions/src/dts.ts at the PARTITIONING_FIELD guard and its comment, then compare the corresponding upstream extension implementation inherited in commit a34ba233. Reproduce the PATCH against a real project in the us location and verify that clearing the field succeeds and changes a subsequent run; done means both implementations allow the empty value without the obsolete error constant.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend, data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.