GoogleCloudPlatform / GoogleCloudPlatform/firebase-extensions
fix(firestore-incremental-capture): array fields do not survive restore - scalars become empty maps, refs/bytes/nulls corrupted
- Dominant language
- TypeScript
- Stars
- 124
- Forks
- 67
- Avg merge
- 5d 5h
- Merged PRs (30d)
- 5
Description
### [REQUIRED] Step 2: Describe your configuration
- Extension name: `firestore-incremental-capture`
- Extension version: 0.0.12 (last published), and the `firestore-incremental-capture-pipeline` Dataflow job it runs
- Configuration values: not configuration dependent - affects every installation that restores documents containing array fields
### [REQUIRED] Step 3: Describe the problem
Array fields do not round-trip through capture and restore. Both sides of the wire format mishandle them:
**Serializer** (`firestore-incremental-capture/functions/src/utils/firestore_serializer.ts`):
- A `null` array item is tagged `{type: 'object', value: null}` because `typeof null === 'object'` (around line 44).
- A `Buffer` or `DocumentReference` inside an array falls into the generic `firestoreSerializer(item)` recursion, which traverses the object's internals into garbage instead of hitting the `binary` / `documentReference` branches.
**Reconstructor** (`firestore-incremental-capture-pipeline/src/main/java/com/pipeline/FirestoreReconstructor.java`):
- `buildFirestoreList` treats every array element as a field map and wraps it in `MapValue`. A tagged scalar element such as `{type: 'string', value: 'x'}` becomes an **empty map**, so arrays of strings, numbers, booleans - any scalar - restore as arrays of empty maps.
#### Steps to reproduce:
1. Install the extension and let the changelog accumulate.
2. After the backup baseline, write a document with `tags: ['a', 'b']` and `refs: [someDocRef, null]`.
3. Run a restoration for a timestamp after the write.
##### Expected result
Arrays restored with their elements intact.
##### Actual result
Scalar arrays restore as arrays of empty maps; reference/buffer/null elements are corrupted or dropped. The job reports success and logs nothing.
#### Impact
Silent data corruption on restore for any document using arrays - one of the most common Firestore field shapes. Restoration is the disaster-recovery path.
#### Suggested fix
Two-sided:
- Serializer: tag array elements with the same type discrimination used for top-level fields (`null`, `binary`, `documentReference`, scalars), so elements carry `{type, value}` like everything else.
- Reconstructor: in `buildFirestoreList`, detect a `{type, value}` element and build the scalar `Value` directly (reusing the switch), falling back to map handling only for genuine map elements.
The serializer change alters the wire format for array elements, so the reconstructor needs to accept both the old and new element shapes for changelogs already in place.
#### Related
Found during adversarial review of #1146. Same silent-loss family as #1138 (id-collision dedupe) and #1144 (dropped documentReference/binary/null at top level); those two fix the top-level cases only. Pre-existing on main, present in both the extension and the kit port.
Contributor guide
Research direction
Read firestore-incremental-capture/functions/src/utils/firestore_serializer.ts and firestore-incremental-capture-pipeline/src/main/java/com/pipeline/FirestoreReconstructor.java, starting with array handling and buildFirestoreList. Reproduce the restore with scalar, reference, buffer, and null array elements, then verify that new and existing changelog shapes restore intact without silent corruption.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, typescript
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100