GoogleCloudPlatform / GoogleCloudPlatform/firebase-extensions

fix(firestore-incremental-capture): array fields do not survive restore - scalars become empty maps, refs/bytes/nulls corrupted

Open
#1,147 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
124
Forks
67
Avg merge
5d 5h
Merged PRs (30d)
5

Description

### [REQUIRED] Step 2: Describe your configuration

- Extension name: `firestore-incremental-capture`
- Extension version: 0.0.12 (last published), and the `firestore-incremental-capture-pipeline` Dataflow job it runs
- Configuration values: not configuration dependent - affects every installation that restores documents containing array fields

### [REQUIRED] Step 3: Describe the problem

Array fields do not round-trip through capture and restore. Both sides of the wire format mishandle them:

**Serializer** (`firestore-incremental-capture/functions/src/utils/firestore_serializer.ts`):

- A `null` array item is tagged `{type: 'object', value: null}` because `typeof null === 'object'` (around line 44).
- A `Buffer` or `DocumentReference` inside an array falls into the generic `firestoreSerializer(item)` recursion, which traverses the object's internals into garbage instead of hitting the `binary` / `documentReference` branches.

**Reconstructor** (`firestore-incremental-capture-pipeline/src/main/java/com/pipeline/FirestoreReconstructor.java`):

- `buildFirestoreList` treats every array element as a field map and wraps it in `MapValue`. A tagged scalar element such as `{type: 'string', value: 'x'}` becomes an **empty map**, so arrays of strings, numbers, booleans - any scalar - restore as arrays of empty maps.

#### Steps to reproduce:

1. Install the extension and let the changelog accumulate.
2. After the backup baseline, write a document with `tags: ['a', 'b']` and `refs: [someDocRef, null]`.
3. Run a restoration for a timestamp after the write.

##### Expected result

Arrays restored with their elements intact.

##### Actual result

Scalar arrays restore as arrays of empty maps; reference/buffer/null elements are corrupted or dropped. The job reports success and logs nothing.

#### Impact

Silent data corruption on restore for any document using arrays - one of the most common Firestore field shapes. Restoration is the disaster-recovery path.

#### Suggested fix

Two-sided:

- Serializer: tag array elements with the same type discrimination used for top-level fields (`null`, `binary`, `documentReference`, scalars), so elements carry `{type, value}` like everything else.
- Reconstructor: in `buildFirestoreList`, detect a `{type, value}` element and build the scalar `Value` directly (reusing the switch), falling back to map handling only for genuine map elements.

The serializer change alters the wire format for array elements, so the reconstructor needs to accept both the old and new element shapes for changelogs already in place.

#### Related

Found during adversarial review of #1146. Same silent-loss family as #1138 (id-collision dedupe) and #1144 (dropped documentReference/binary/null at top level); those two fix the top-level cases only. Pre-existing on main, present in both the extension and the kit port.

Contributor guide

Open the contributing guide

Research direction

Read firestore-incremental-capture/functions/src/utils/firestore_serializer.ts and firestore-incremental-capture-pipeline/src/main/java/com/pipeline/FirestoreReconstructor.java, starting with array handling and buildFirestoreList. Reproduce the restore with scalar, reference, buffer, and null array elements, then verify that new and existing changelog shapes restore intact without silent corruption.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, typescript
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.