firebase / firebase/extensions
[firestore-bigquery-export] v0.2.6 YAML parsing error with multi-wildcard collection paths
- Dominant language
- TypeScript
- Stars
- 979
- Forks
- 433
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 127
Description
# Bug Report: firestore-bigquery-export v0.2.6 - Wildcard Paths Cause YAML Parsing Error
## Bug Information
**Extension:** `firebase/firestore-bigquery-export`
**Broken Version:** v0.2.6
**Working Version:** v0.1.56
**Severity:** Critical - Breaks multi-tenant architectures using wildcard paths
**Status:** Regression introduced in v2 function migration
---
## Summary
The `firestore-bigquery-export` extension v0.2.6 fails to deploy when `COLLECTION_PATH` contains multiple wildcards (e.g., `{org_id}/{client_id}/data/call_records/data`). This is a regression from v0.1.56 which handled wildcard paths correctly.
The error occurs during Google Cloud Deployment Manager YAML parsing, caused by unquoted curly braces in the v2 function eventFilters configuration.
---
## Environment
- **Firebase CLI Version:** 13.x
- **Project Type:** Multi-region (nam5)
- **Extension Version Tested:** v0.2.6
- **Extension Version Working:** v0.1.56
- **GCP Project:** Multiple projects tested (dev, demo, prod)
---
## Reproduction Steps
### 1. Configure Extension with Multi-Wildcard Path
Create an extension configuration with a collection path containing multiple wildcards:
**firebase.json:**
```json
{
"extensions": {
"firestore-bigquery-export-callinsights": "firebase/firestore-bigquery-export@0.2.6"
}
}
```
**extensions/firestore-bigquery-export-callinsights.env:**
```env
BIGQUERY_PROJECT_ID=${param:PROJECT_ID}
DATABASE_REGION=nam5
COLLECTION_PATH={org_id}/{client_id}/data/call_records/data
DATASET_ID=my_dataset
DATASET_LOCATION=us
TABLE_ID=call_insights
WILDCARD_IDS=true
```
### 2. Deploy Extension
```bash
firebase deploy --only extensions --force
```
### 3. Observe Error
```
Extensions deploy had errors:
- update firestore-bigquery-export-callinsights
failed to startDeploymentManagerWork for instance id: firestore-bigquery-export-callinsights
error updating deployment: DM HTTP 412 error
Error parsing configuration: while parsing a block mapping
in 'string', line 71, column 19:
- attribute: document
^
expected , but found ''
in 'string', line 73, column 34:
value: {org_id}/{client_id}/data/call_records/data/{documentId}
^
```
---
## Expected Behavior
The extension should deploy successfully with multi-wildcard collection paths, as it did in v0.1.56.
**Working in v0.1.56:**
- ✅ Single wildcard paths: `posts/{postId}/comments`
- ✅ Multi-wildcard paths: `{org_id}/{client_id}/data/call_records/data`
- ✅ All 16 instances deployed successfully
---
## Actual Behavior
**Broken in v0.2.6:**
- ✅ Single wildcard paths: `posts/{postId}/comments` (works)
- ❌ Multi-wildcard paths: `{org_id}/{client_id}/data/call_records/data` (YAML parsing error)
- ❌ 9 out of 16 instances failed to deploy in our test
- ❌ Extensions left in ERRORED state
---
## Root Cause Analysis
### v0.1.56 Implementation (Working)
```yaml
resources:
- name: fsexportbigquery
type: firebaseextensions.v1beta.function
properties:
eventTrigger:
eventType: providers/cloud.firestore/eventTypes/document.write
resource: projects/${param:PROJECT_ID}/databases/(default)/documents/${param:COLLECTION_PATH}/{documentId}
```
- Uses v1 Cloud Functions
- `COLLECTION_PATH` is substituted into a simple resource string
- No YAML parsing of the path value
- Works with any number of curly braces
### v0.2.6 Implementation (Broken)
```yaml
resources:
- name: fsexportbigquery
type: firebaseextensions.v1beta.v2function
properties:
eventTrigger:
eventFilters:
- attribute: document
value: ${COLLECTION_PATH}/{documentId} # ← UNQUOTED!
operator: match-path-pattern
```
- Uses v2 Cloud Functions
- `value` field is **not quoted**
- After substitution: `value: {org_id}/{client_id}/data/call_records/data/{documentId}`
- YAML parser interprets `{org_id}` as start of a flow mapping (object)
- Fails parsing when it encounters `/` after the opening brace
### YAML Parsing Issue
When the Deployment Manager tries to parse the generated YAML:
```yaml
eventFilters:
- attribute: document
value: {org_id}/{client_id}/data/call_records/data/{documentId}
```
YAML parser sees:
- `{org_id}` - Start of a mapping object
- `/` - Invalid inside a mapping
- **Error:** Expected closing `}` or key-value pair, found `/`
---
## Proposed Fix
### Option 1: Quote the Value (Recommended)
```yaml
eventFilters:
- attribute: document
value: "${COLLECTION_PATH}/{documentId}" # ← Add quotes
operator: match-path-pattern
```
### Option 2: Use YAML Literal String
```yaml
eventFilters:
- attribute: document
value: |-
${COLLECTION_PATH}/{documentId}
operator: match-path-pattern
```
### Option 3: Escape in Template Generation
If the template is generated programmatically, escape the curly braces or quote the entire value before writing to YAML.
---
## Impact
### Who Is Affected
1. **Multi-tenant architectures** using org/client ID wildcards
2. **Anyone with multiple wildcards** in collection paths
3. **Existing v0.1.56 users** attempting to upgrade
### Scope
- **Our testing:** 9 out of 16 extensions failed (56% failure rate)
- **Pattern:** All failures had paths like `{x}/{y}/...`
- **Pattern:** All successes had simple paths or single wildcards
### Business Impact
- ❌ Cannot upgrade to v0.2.6
- ❌ Missing out on v2 functions benefits (Node.js 20, better performance)
- ❌ Stuck on v0.1.56 indefinitely
- ❌ Multi-tenant SaaS platforms blocked from upgrading
---
## Workarounds
### Current Workaround: Stay on v0.1.56
```json
{
"extensions": {
"firestore-bigquery-export-callinsights": "firebase/firestore-bigquery-export@0.1.56"
}
}
```
**Pros:**
- ✅ Stable and production-proven
- ✅ Full wildcard support
**Cons:**
- ❌ Uses deprecated v1 Cloud Functions
- ❌ Node.js 18 (older runtime)
- ❌ Missing new features from v0.2.x
### No Alternative Workaround
- Cannot modify extension source (it's published)
- Cannot manually deploy v2 functions (managed by extension)
- Must wait for Firebase to fix and publish v0.2.7
---
## Test Cases
### Should Pass (Regression Tests)
```
✅ COLLECTION_PATH=posts
✅ COLLECTION_PATH=users/{userId}/posts
✅ COLLECTION_PATH=orgs/{orgId}/data
❌ COLLECTION_PATH={orgId}/{clientId}/data (FAILS in v0.2.6)
❌ COLLECTION_PATH={a}/{b}/c/{d}/data (FAILS in v0.2.6)
❌ COLLECTION_PATH={org_id}/{client_id}/voice_assistant_app/data/chats (FAILS in v0.2.6)
```
### Edge Cases to Test
```
COLLECTION_PATH={a}/{b}/{c}/data
COLLECTION_PATH={tenant}/{workspace}/{project}/collections/{collectionId}/documents
COLLECTION_PATH=customers/{customerId}/sites/{siteId}/devices/{deviceId}/readings
```
---
## Additional Context
### Related Issues
- Wildcard support is documented: https://firebase.google.com/docs/extensions/official/firestore-bigquery-export
- Validation regex still allows wildcards: `^[^/]+(/[^/]+/[^/]+)*$`
- No deprecation notice in CHANGELOG.md for v0.2.0
- Issue #484: "Firestore BigQuery Export: The backfill script doesn't support wildcards" (different but related)
### Changelog Entry (v0.2.0)
```
## Version 0.2.0
- feat - migrate to v2 functions and support non-default firestore instances
- fix - fix enqueue logic and types
```
No mention of breaking changes to wildcard support.
### Version Comparison
| Feature | v0.1.56 | v0.2.6 |
|---------|---------|--------|
| Wildcard paths | ✅ Works | ❌ Broken |
| Multi-wildcard paths | ✅ Works | ❌ Broken |
| v2 Functions | ❌ No | ✅ Yes |
| Node.js 20 | ❌ No | ✅ Yes |
| DATABASE_REGION param | ❌ No | ✅ Yes (undocumented) |
---
## Reproduction Project
We can provide:
- Full extension configuration files
- Deployment logs
- Multiple test cases
- Before/after comparisons
**Available for testing/debugging upon request.**
---
## Suggested Verification
After fix is implemented, please verify:
1. ✅ Single wildcard paths still work
2. ✅ Multi-wildcard paths work
3. ✅ Paths with 3+ wildcards work
4. ✅ Special characters in wildcard names (underscore, dash)
5. ✅ Migration path from v0.1.56 → v0.2.x works smoothly
6. ✅ Existing deployed extensions can be updated in-place
---
## Timeline
- **v0.1.56 Released:** January 2025 (stable)
- **v0.2.0 Released:** Unknown (migration to v2 functions)
- **v0.2.6 Current:** Latest version (broken)
- **Bug Discovered:** October 2, 2025
- **Environments Affected:** Dev (recovered), Demo (stable on v0.1.56), Prod (stable on v0.1.56)
---
## Request
Please:
1. ✅ Confirm this is a bug (not intentional removal of feature)
2. ✅ Add regression tests for multi-wildcard paths
3. ✅ Fix by quoting the `value` field in eventFilters
4. ✅ Document DATABASE_REGION parameter (currently undocumented)
5. ✅ Publish v0.2.7 with fix
6. ✅ Update documentation if any limitations exist
---
## Contact
Available for:
- Further testing
- Providing detailed logs
- Testing pre-release versions
- Verification of fix
---
**Thank you for maintaining this excellent extension!** We rely on it heavily for our multi-tenant SaaS platform and would love to upgrade to v2 functions once this is resolved.
Contributor guide
Assessment
This issue has not been assessed yet.