firebase / firebase/firebase-tools
Support regional secrets for extensions deploy
- Dominant language
- TypeScript
- Stars
- 4.5k
- Forks
- 1.3k
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 84
Description
### [REQUIRED] Environment info
**firebase-tools:** 12.7.0
**Platform:** macOS
### [REQUIRED] Test case
- Enable the Ory Policy `constraints/gcp.resourceLocations`
- Try to deploy an extension with a secret stored in Secret Manager
### [REQUIRED] Steps to reproduce
- Go to https://console.cloud.google.com/iam-admin/orgpolicies/gcp-resourceLocations
- Restrict the location to a specific subset (e.g. allowed `in:eu-locations`)
- Run `firebase ext:install elastic/firestore-elastic-app-search --project=projectId_or_alias`
### [REQUIRED] Expected behavior
The extension installation succeeds successfully, e.g. by only putting the secret in the region of the deployed Cloud Function (instead of global)
### [REQUIRED] Actual behavior
The deployment fails with the error:
```
Error: HTTP Error: 400, Constraint constraints/gcp.resourceLocations violated for [orgpolicy:projects/123456789] attempting to create a secret in [global]. For more information, see https://cloud.google.com/resource-manager/docs/organization-policy/defining-locations.
```
Full logs
```
[debug] [2023-10-17T11:31:10.823Z] ----------------------------------------------------------------------
[debug] [2023-10-17T11:31:10.824Z] Command: /Users/dennis/.nvm/versions/node/v16.18.1/bin/node /Users/dennis/.yarn/bin/firebase ext:install . --project=not-a-project --debug
[debug] [2023-10-17T11:31:10.824Z] CLI Version: 12.7.0
[debug] [2023-10-17T11:31:10.825Z] Platform: darwin
[debug] [2023-10-17T11:31:10.825Z] Node Version: v16.18.1
[debug] [2023-10-17T11:31:10.825Z] Time: Tue Oct 17 2023 11:31:10 GMT+0000 (Coordinated Universal Time)
[debug] [2023-10-17T11:31:10.826Z] Env Overrides: FIREBASE_FUNCTIONS_DEFAULT_REGION
[debug] [2023-10-17T11:31:10.826Z] ----------------------------------------------------------------------
[debug]
[debug] [2023-10-17T11:31:10.860Z] > command requires scopes: ["email","openid","https://www.googleapis.com/auth/cloudplatformprojects.readonly","https://www.googleapis.com/auth/firebase","https://www.googleapis.com/auth/cloud-platform"]
[debug] [2023-10-17T11:31:10.861Z] > authorizing via signed-in user (user@example.com)
[debug] [2023-10-17T11:31:10.861Z] [iam] checking project not-a-project for permissions ["firebase.projects.get","firebaseextensions.instances.create"]
[debug] [2023-10-17T11:31:10.862Z] >>> [apiv2][query] POST https://cloudresourcemanager.googleapis.com/v1/projects/not-a-project:testIamPermissions [none]
[debug] [2023-10-17T11:31:10.862Z] >>> [apiv2][(partial)header] POST https://cloudresourcemanager.googleapis.com/v1/projects/not-a-project:testIamPermissions x-goog-quota-user=projects/not-a-project
[debug] [2023-10-17T11:31:10.862Z] >>> [apiv2][body] POST https://cloudresourcemanager.googleapis.com/v1/projects/not-a-project:testIamPermissions {"permissions":["firebase.projects.get","firebaseextensions.instances.create"]}
[debug] [2023-10-17T11:31:11.585Z] <<< [apiv2][status] POST https://cloudresourcemanager.googleapis.com/v1/projects/not-a-project:testIamPermissions 200
[debug] [2023-10-17T11:31:11.585Z] <<< [apiv2][body] POST https://cloudresourcemanager.googleapis.com/v1/projects/not-a-project:testIamPermissions {"permissions":["firebase.projects.get","firebaseextensions.instances.create"]}
[info] i extensions: ensuring required API firebaseextensions.googleapis.com is enabled...
[debug] [2023-10-17T11:31:11.586Z] >>> [apiv2][query] GET https://serviceusage.googleapis.com/v1/projects/not-a-project/services/firebaseextensions.googleapis.com [none]
[debug] [2023-10-17T11:31:11.587Z] >>> [apiv2][(partial)header] GET https://serviceusage.googleapis.com/v1/projects/not-a-project/services/firebaseextensions.googleapis.com x-goog-quota-user=projects/not-a-project
[debug] [2023-10-17T11:31:12.576Z] <<< [apiv2][status] GET https://serviceusage.googleapis.com/v1/projects/not-a-project/services/firebaseextensions.googleapis.com 200
[debug] [2023-10-17T11:31:12.576Z] <<< [apiv2][body] GET https://serviceusage.googleapis.com/v1/projects/not-a-project/services/firebaseextensions.googleapis.com [omitted]
[info] ✔ extensions: required API firebaseextensions.googleapis.com is enabled
[debug] [2023-10-17T11:31:12.579Z] >>> [apiv2][query] GET https://firebase.googleapis.com/v1beta1/projects/not-a-project [none]
[debug] [2023-10-17T11:31:12.822Z] <<< [apiv2][status] GET https://firebase.googleapis.com/v1beta1/projects/not-a-project 200
[debug] [2023-10-17T11:31:12.822Z] <<< [apiv2][body] GET https://firebase.googleapis.com/v1beta1/projects/not-a-project {"projectId":"not-a-project","projectNumber":"1234567890","displayName":"not-a-project","name":"projects/not-a-project","resources":{"hostingSite":"not-a-project-46a13","storageBucket":"not-a-project.appspot.com","locationId":"europe-west"},"state":"ACTIVE","etag":"1_51ebbee2-5413-4433-9d20-5b32e6afd29e"}
[info] i extensions: Checking project IAM policy...
[debug] [2023-10-17T11:31:12.824Z] >>> [apiv2][query] GET https://firebaseextensions.googleapis.com/v1beta/projects/not-a-project/instances pageSize=100&pageToken=
[debug] [2023-10-17T11:31:14.387Z] <<< [apiv2][status] GET https://firebaseextensions.googleapis.com/v1beta/projects/not-a-project/instances 200
[debug] [2023-10-17T11:31:21.564Z] >>> [apiv2][query] POST https://cloudresourcemanager.googleapis.com/v1/projects/not-a-project:getIamPolicy [none]
[debug] [2023-10-17T11:31:22.485Z] <<< [apiv2][status] POST https://cloudresourcemanager.googleapis.com/v1/projects/not-a-project:getIamPolicy 200
[debug] [2023-10-17T11:31:14.392Z] <<< [apiv2][body] GET https://firebaseextensions.googleapis.com/v1beta/projects/not-a-project/instances {"instances":[]}
[debug] [2023-10-17T11:31:22.486Z] <<< [apiv2][body] POST https://cloudresourcemanager.googleapis.com/v1/projects/not-a-project:getIamPolicy {"version":1,"etag":"BwYH0WV7oDw=","bindings":[]}
[debug] {"version":1,"etag":"BwYH0WV7oDw=","bindings":[]}
[info] ✔ extensions: Project IAM policy OK
[debug] [2023-10-17T11:31:22.638Z] Archived 402.57 KB in ..
[debug] [2023-10-17T11:31:22.639Z] >>> [apiv2][query] PUT https://storage.googleapis.com/firebase-ext-eap-uploads/firebase-archive--58488-i3qPvDbbgHO0-.zip [none]
[debug] [2023-10-17T11:31:22.639Z] >>> [apiv2][body] PUT https://storage.googleapis.com/firebase-ext-eap-uploads/firebase-archive--58488-i3qPvDbbgHO0-.zip [stream]
[debug] [2023-10-17T11:31:23.501Z] <<< [apiv2][status] PUT https://storage.googleapis.com/firebase-ext-eap-uploads/firebase-archive--58488-i3qPvDbbgHO0-.zip 200
[debug] [2023-10-17T11:31:23.501Z] <<< [apiv2][body] PUT https://storage.googleapis.com/firebase-ext-eap-uploads/firebase-archive--58488-i3qPvDbbgHO0-.zip undefined
[debug] [2023-10-17T11:31:23.502Z] >>> [apiv2][query] POST https://firebaseextensions.googleapis.com/v1beta/projects/not-a-project/sources/ [none]
[debug] [2023-10-17T11:31:23.502Z] >>> [apiv2][body] POST https://firebaseextensions.googleapis.com/v1beta/projects/not-a-project/sources/ {"packageUri":"https://storage.googleapis.com/firebase-ext-eap-uploads/firebase-archive--58488-i3qPvDbbgHO0-.zip?alt=media","extensionRoot":"/"}
[debug] [2023-10-17T11:31:24.359Z] <<< [apiv2][status] POST https://firebaseextensions.googleapis.com/v1beta/projects/not-a-project/sources/ 200
[debug] [2023-10-17T11:31:24.359Z] <<< [apiv2][body] POST https://firebaseextensions.googleapis.com/v1beta/projects/not-a-project/sources/ {"name":"projects/not-a-project/operations/71fad467-3c6e-43c9-bccc-63dac47c8646","metadata":{"@type":"type.googleapis.com/google.firebase.extensions.v1beta.ExtensionSource","name":"projects/not-a-project/sources/aae91f2b-6917-4b50-a845-d5b64f75591c","packageUri":"https://storage.googleapis.com/firebase-ext-eap-uploads/firebase-archive--58488-i3qPvDbbgHO0-.zip?alt=media","extensionRoot":"/","lastOperationName":"projects/not-a-project/operations/71fad467-3c6e-43c9-bccc-63dac47c8646"}}
[debug] [2023-10-17T11:31:24.361Z] >>> [apiv2][query] GET https://firebaseextensions.googleapis.com/v1beta/projects/not-a-project/operations/71fad467-3c6e-43c9-bccc-63dac47c8646 [none]
[debug] [2023-10-17T11:31:25.512Z] <<< [apiv2][status] GET https://firebaseextensions.googleapis.com/v1beta/projects/not-a-project/operations/71fad467-3c6e-43c9-bccc-63dac47c8646 200
[debug] [2023-10-17T11:31:25.512Z] <<< [apiv2][body] GET https://firebaseextensions.googleapis.com/v1beta/projects/not-a-project/operations/71fad467-3c6e-43c9-bccc-63dac47c8646 {"name":"projects/not-a-project/operations/71fad467-3c6e-43c9-bccc-63dac47c8646","done":true,"response":{"@type":"type.googleapis.com/google.firebase.extensions.v1beta.ExtensionSource","name":"projects/not-a-project/sources/aae91f2b-6917-4b50-a845-d5b64f75591c","packageUri":"https://storage.googleapis.com/firebase-ext-eap-uploads/firebase-archive--58488-i3qPvDbbgHO0-.zip?alt=media","hash":"df60381b4ad8e99307d2d635e4aa8b9a5ddab605ec9a35e226a556472817b732","extensionRoot":"/","spec":{"specVersion":"v1beta","name":"firestore-elastic-app-search","version":"0.4.1","description":"Syncs documents from a Firestore collection to Elastic App Search to enable full-text search.","roles":[{"role":"datastore.user","reason":"Allows the extension to read configuration and build bundles from Firestore."}],"resources":[{"name":"shipToElastic","type":"firebaseextensions.v1beta.function","propertiesYaml":"eventTrigger:\n eventType: providers/cloud.firestore/eventTypes/document.write\n resource: projects/${PROJECT_ID}/databases/(default)/documents/${COLLECTION_PATH}/{documentId}\nlocation: ${LOCATION}\nruntime: nodejs14\n","description":"Function triggered on Create, Update, or Delete of a document in the specified collection which syncs the change to App Search.","deletionPolicy":"DELETE"}],"billingRequired":true,"author":{"authorName":"Elastic","url":"https://www.elastic.co/"},"license":"Apache-2.0","sourceUrl":"https://github.com/elastic/app-search-firestore-extension","params":[{"param":"LOCATION","label":"Cloud Functions location","type":"SELECT","description":"Choose where you want to deploy the functions created for this extension. For help selecting a location, refer to the [location selection guide](https://firebase.google.com/docs/functions/locations).","required":true,"options":[{"value":"us-central1","label":"Iowa (us-central1)"},{"value":"us-east1","label":"South Carolina (us-east1)"},{"value":"us-east4","label":"Northern Virginia (us-east4)"},{"value":"us-west2","label":"Los Angeles (us-west2)"},{"value":"us-west3","label":"Salt Lake City (us-west3)"},{"value":"us-west4","label":"Las Vegas (us-west4)"},{"value":"europe-central2","label":"Warsaw (europe-central2)"},{"value":"europe-west1","label":"Belgium (europe-west1)"},{"value":"europe-west2","label":"London (europe-west2)"},{"value":"europe-west3","label":"Frankfurt (europe-west3)"},{"value":"europe-west6","label":"Zurich (europe-west6)"},{"value":"asia-east2","label":"Hong Kong (asia-east2)"},{"value":"asia-northeast1","label":"Tokyo (asia-northeast1)"},{"value":"asia-northeast2","label":"Osaka (asia-northeast2)"},{"value":"asia-northeast3","label":"Seoul (asia-northeast3)"},{"value":"asia-south1","label":"Mumbai (asia-south1)"},{"value":"asia-southeast2","label":"Jakarta (asia-southeast2)"},{"value":"northamerica-northeast1","label":"Montreal (northamerica-northeast1)"},{"value":"southamerica-east1","label":"Sao Paulo (southamerica-east1)"},{"value":"australia-southeast1","label":"Sydney (australia-southeast1)"}],"immutable":true},{"param":"COLLECTION_PATH","label":"Collection path","type":"STRING","description":"The path to the collection that you want to sync to App Search.\n","required":true,"example":"movies","validationRegex":"^[^/]+(/[^/]+/[^/]+)*$","validationErrorMessage":"Must be a valid Cloud Firestore Collection"},{"param":"APP_SEARCH_ENGINE_NAME","label":"Elastic App Search engine name","type":"STRING","description":"The name of the Elastic App Search \"engine\" you want to sync to your collection.\n","required":true,"example":"movies"},{"param":"APP_SEARCH_API_KEY","label":"Elastic App Search private API key","type":"SECRET","description":"A \"private\" API key from your Elastic App Search deployment with access to the engine named above. API keys can be found in the [App Search \"Credentials\" page](https://www.elastic.co/guide/en/app-search/current/authentication.html#authentication-api-keys).","required":true,"example":"private-79iadc5dzd3qxgfgd9w9ryc7"},{"param":"ENTERPRISE_SEARCH_URL","label":"Elastic Enterprise Search URL","type":"STRING","description":"The base URL of your Enterprise Search deployment. This can also be found in the [App Search \"Credentials\" page](https://www.elastic.co/guide/en/app-search/current/authentication.html#authentication-api-keys).\n","required":true,"example":"https://example.ent.us-west1.gcp.cloud.es.io/"},{"param":"INDEXED_FIELDS","label":"Indexed fields","type":"STRING","description":"A comma separated list of the fields to index from your collection. Only the fields listed will be synced and searchable in App Search. You can specify [fields nested](https://github.com/elastic/app-search-firestore-extension/blob/master/POSTINSTALL.md#nested-fields) in maps with underscores: `field__subField`, and [new names for fields](https://github.com/elastic/app-search-firestore-extension/blob/master/POSTINSTALL.md#field-name-compatibility-and-renaming) in App Search using a double colon: `previousname::newname`.","required":true,"example":"producer,director__name,year"}],"preinstallContent":"\n\nThe Elastic App Search Firestore extension enables comprehensive [full-text search](https://firebase.google.com/docs/firestore/solutions/search) for your Firebase applications.\n\nThis extension indexes and syncs the documents in a Cloud Firestore collection to an [Elastic App Search](https://www.elastic.co/app-search?ultron=firebase-extension&blade=preinstall&hulk=product) deployment by creating a Cloud Function which syncs changes in your collection on any [write event](https://firebase.google.com/docs/functions/firestore-events#function_triggers) (any time you create, update, or a delete a document).\n\n#### Elastic App Search\n\nElastic App Search provides a comprehensive API for implementing common search patterns like auto-completed search suggestions and faceted filter navigation. You'll also have tooling so your team can easily track and tweak search relevance based on usage data.\n\nApp Search is a part of [Elastic Enterprise Search](https://www.elastic.co/guide/en/enterprise-search/current/installation.html). You'll need an Enterprise Search deployment, which is created and maintained outside of Firebase. \n\n#### Getting started\n\n1. Start an Enterprise Search deployment. You can provision one easily with [Elastic Cloud on GCP](https://console.cloud.google.com/marketplace/product/endpoints/elasticsearch-service.gcpmarketplace.elastic.co).\n2. Once you have a deployment running, you'll need an [App Search Engine](https://www.elastic.co/guide/en/app-search/current/getting-started.html#getting-started-with-app-search-engine) to sync to your collection.\n3. Once you've installed the extension and your Firestore collection is synced to App Search, you're ready to [start searching](https://www.elastic.co/guide/en/app-search/current/search-guide.html)!\n\nYou can use the App Search [Search API](https://www.elastic.co/guide/en/app-search/current/search.html) for full-text search and everything you need to build a complete search experience: facets, filters, click analytics, query suggestion, relevance tuning and much more.\n\nIf you have documents in your collection already, this extension also provides a [script](https://github.com/elastic/app-search-firestore-extension/tree/master/functions/src/bin) for backfilling existing data to App Search.\n\n\n\n#### Billing\n \nTo install an extension, your project must be on the [Blaze (pay as you go) plan](https://firebase.google.com/pricing)\n \n- You will be charged a small amount (typically around $0.01/month) for the Firebase resources required by this extension (even if it is not used).\n- This extension uses other Firebase and Google Cloud Platform services, which have associated charges if you exceed the service’s free tier:\n - Cloud Functions (Node.js 10+ runtime. See [FAQs](https://firebase.google.com/support/faq#expandable-24))\n - Cloud Firestore\n - Cloud Secret Manager\n\nIf you host your Elastic Enterprise Search instance on Elastic Cloud, you will also be responsible for charges associated with that service.\n\n[Learn more about Elastic Cloud](https://www.elastic.co/cloud?ultron=firebase-extension&blade=preinstall&hulk=product).\n","postinstallContent":"\n\n### See it in action\n\nYou can test out this extension right away!\n\n1. Go to your [Cloud Firestore dashboard](https://console.firebase.google.com/project/${param:PROJECT_ID}/firestore/data) in the Firebase console.\n\n2. If it doesn't already exist, create the collection you specified during installation: `${param:COLLECTION_PATH}`\n\n3. Create a document in the collection that contains any of the fields you specified as indexed fields during installation:\n\n```js\n`${param:INDEXED_FIELDS}`\n```\n\n4. Go to the documents page of the Engine you created inside of your [App Search Dashboard](${param:ENTERPRISE_SEARCH_URL}/as#/engines/${param:APP_SEARCH_ENGINE_NAME}/documents). You should see the that document you just created listed on this page.\n\n### Using the extension\n\nWhenever a document is created, updated, imported, or deleted in the specified collection, this extension sends that update to App Search. You can then run tull-text searches on this mirrored dataset.\n\nAfter documents are indexed into App Search, you'll have the complete App Search [Search API](https://www.elastic.co/guide/en/app-search/current/search.html) available to you for searching.\n\nNote that this extension only listens for document changes in the collection, but not changes in any subcollection.\n\n### _(Optional)_ Backfill or import existing documents\n\nThis extension only sends the content of documents that have been changed -- it does not export your full dataset of existing documents into App Search. So, to backfill your dataset with all the documents in your collection, you can run the import script provided by this extension.\n\nBefore running the script, first follow the instructions [here](https://firebase.google.com/docs/admin/setup#initialize-sdk) to \"To generate a private key file for your service account\". Download it and save it somewhere as `serviceAccountKey.json`.\n\n```shell\nGOOGLE_APPLICATION_CREDENTIALS= /path/to/your/serviceAccountKey.json \\\nCOLLECTION_PATH=${param:COLLECTION_PATH} \\\nINDEXED_FIELDS=${param:INDEXED_FIELDS} \\\nENTERPRISE_SEARCH_URL=${param:ENTERPRISE_SEARCH_URL} \\\nAPP_SEARCH_API_KEY= { your private app search API key here } \\\nAPP_SEARCH_ENGINE_NAME=${param:APP_SEARCH_ENGINE_NAME} \\\nnpx @elastic/app-search-firestore-extension import\n```\n\n### _(Optional)_ Configure App Search engine schema\n\nIt is important to note that all data is initially indexed into App Search as text fields.\n\nThis means that even if your field is a `timestamp` or `number` in Firestore, it will be indexed as text in App Search initially.\n\nThis is fine for fields that you'd like to perform full-text search on. However, if you plan to do something like sort numerically or implement range filters when calling `search`, you should first visit the Schema page for your Engine in the App Search Dashboard and select the correct types for your fields.\n\nYou can read more about Schemas [here](https://www.elastic.co/guide/en/app-search/current/indexing-documents-guide.html#indexing-documents-guide-schema).\n\n### _(Optional)_ Reindex\n\nThere may be times where you want to reindex all of your documents from this collection to App Search.\n\nFor instance, if you change the \"indexed fields\" configuration in this extension, you should then run a reindex in order to make sure that the changes are picked up in App Search.\n\nTo reindex data, use the steps listed above for \"Backfill or import existing documents\".\n\n### How documents are indexed in App Search\n\nThe TLDR for this section is:\n\n- `text` and `number` type fields are indexed as-is to App Search.\n- `geo` and `timestamp` fields are formatted slightly differently when indexed.\n- `map`, `boolean`, and `reference` are not supported by App Search, they will be indexed as text.\n- nested arrays are not supported at all and will be dropped before indexing in App Search.\n- While `map`s are not supported, you _can_ specify that fields within a map get indexed as top level fields in App Search, using the `__` syntax when configuring indexed fields\n- App Search only supports lower-cased alphanumeric characters and underscores (\"\\_\") in field names. This extension will rename fields that don't match, or you can use the `::` syntax to specify what it is renamed to when configuring indexed fields.\n\nIt is important to note that not all [data types supported by Firestore](https://firebase.google.com/docs/firestore/manage-data/data-types) are compatible with the [data types supported by App Search](https://www.elastic.co/guide/en/app-search/current/api-reference.html#overview-api-references-schema-design).\n\nSome types are supported in a 1-to-1 way: `text`, `number`.\n\nOthers are supported, but formatted slightly differently: `timestamp`, `geo`.\n\nOthers are simply not supported: `boolean`, `map`, `reference`.\n\n**It is also important to note that ONLY fields that you specify as Indexed Fields in this extension will be indexed into App Search**.\n\nFor example, given the following document in Firestore:\n\n```json\n{\n \"id\": \"12345\",\n \"name\": \"Rocky Mountain\",\n \"nps_link\": \"https://www.nps.gov/romo/index.htm\",\n \"states\": [\"Colorado\"],\n \"visitors\": 4517585,\n \"world_heritage_site\": false,\n \"location\": {\n \"_latitude\": 41.12,\n \"_longitude\": -71.34\n },\n \"acres\": 265795.2,\n \"square_km\": 1075.6,\n \"date_established\": {\n \"_seconds\": 1631213624,\n \"_nanoseconds\": 176000000\n }\n}\n```\n\nIf you've configured the plugin with indexed fields of `name,states`, then the document will be indexed into App Search as the following:\n\n```json\n{\n \"id\": \"12345\",\n \"name\": \"Rocky Mountain\",\n \"states\": [\"Colorado\"]\n}\n```\n\nThat means you could then perform a search with the App Search Search API over the `name` and `states` fields for results.\n\n#### Similar types, formatted differently\n\nAs mentioned above, types are somtimes formatted differently in App Search. So given the same example document above, but configured with `name,states,location,date_established` as the indexed fields, you'll see that the `location` and `date_established` fields have been formatted slightly differently.\n\n```json\n{\n \"id\": \"12345\",\n \"name\": \"Rocky Mountain\",\n \"states\": [\"Colorado\"],\n \"location\": \"41.12,-71.34\",\n \"date_established\": \"2021-09-09T18:53:44.000Z\"\n}\n```\n\nWe put them in this special format so that App Search is able to recognize them as the correct types. Unlike the name and states fields, you may want to do more than just searching on these fields. In fact, you most likely won't want to search on these fields at all; it's much more likely that you'll want to use these for things like filtering and sorting.\n\n#### Types not supported by App Search\n\nThere are some types of fields that ARE supported by Firestore, but not by App Search. So, when data is indexed, you may see that some data is dropped, or see that it is indexed in a way you may not have expected.\n\n**Maps**\n\nApp Search does not support the concepts of maps. You may only have top-level fields. If you send a map to App Search, it will simply serialize your map and store it as a text:\n\nFirestore:\n\n```json\n{\n \"id\": \"12345\",\n \"foo\": {\n \"bar\": {\n \"baz\": \"some value\"\n }\n }\n}\n```\n\nApp Search:\n\n```json\n{\n \"id\": \"12345\",\n \"foo\": {\n \"bar\": \"{\\\"bar\\\":{\\\"baz\\\":\\\"some value\\\"}}\"\n }\n}\n```\n\n**So are values in maps searchable? Yes, see the Nested Fields section for more info.**\n\n**Nested arrays**\n\nNested arrays are not supported by App Search. Nested arrays will simply be dropped.\n\nFirestore:\n\n```json\n{\n \"id\": \"12345\",\n \"foo\": [[\"a\"]]\n}\n```\n\nApp Search:\n\n```json\n{\n \"id\": \"12345\",\n \"foo\": []\n}\n```\n\n**Reference**\n\nIf you try to index a `reference` field to App Search, it will simply be serialized as if it were any other object, as serialized text:\n\nFirestore:\n\n```json\n{\n \"id\": \"12345\",\n \"some_reference\": {\n \"_firestore\": {\n \"projectId\": \"national_parks\"\n },\n \"_path\": {\n \"segments\": [\"national_parks\", \"123\"]\n },\n \"_converter\": {}\n }\n}\n```\n\nApp Search:\n\n```json\n{\n \"id\": \"12345\",\n \"some_reference\": \"{\\\"_firestore\\\":{\\\"projectId\\\":\\\"national_parks\\\"},\\\"_path\\\":{\\\"segments\\\":[\\\"national_parks\\\",\\\"123\\\"]},\\\"_converter\\\":{}}\"\n}\n```\n\n#### Nested fields\n\nWhile the `map` type is not supported in App Search, you _can_ index fields from within a `map` into App Search. It will convert them to a new top-level field.\n\nIn the provided example, if you used underscores to specify a sub field as indexed, it will index as follows into App Search.\n\nIndexed field: `name,foo__bar__baz`\n\nFirestore:\n\n```json\n{\n \"id\": \"12345\",\n \"name\": \"test name\",\n \"foo\": {\n \"bar\": {\n \"baz\": \"some value\"\n }\n }\n```\n\nApp Search:\n\n```json\n{\n \"id\": \"12345\",\n \"name\": \"test name\",\n \"foo__bar__baz\": \"some value\"\n}\n```\n\nPlease note that we are adding an additional top name field to your schema, in which we use \"\\_\\_\" as a delimiter. This could potentially conflict with other top-level field names, though that will most likely not be the case.\n\n#### Field name compatibility and renaming\n\nApp Search only supports lower-cased alphanumeric characters and underscores (\"\\_\") in field names. Field values that do not match will be renamed to match:\n\nIndexed field: `name,foo__bar__baz`\n\nFirestore:\n\n```json\n{\n \"id\": \"12345\",\n \"a大\": \"a大\",\n \"A1-b-c\": \"A1-b-c\",\n \"d e_f\": \"d e_f\",\n \"大\": \"大\"\n}\n```\n\nApp Search:\n\n```json\n{\n \"id\": \"12345\",\n \"a\": \"a大\",\n \"a1bc\": \"A1-b-c\",\n \"de_f\": \"d e_f\"\n // 大 is ommited entirely because it serialized to an empty string\n}\n```\n\nAs this could have underirable effects we allow renaming of fields by using the `::` when specifying indexed fields:\n\nIndexed field: `你好::hello,爱::love,幸福::happiness`\n\nFirestore:\n\n```json\n{\n \"id\": \"12345\",\n \"你好\": \"你好\",\n \"爱\": \"爱\",\n \"幸福\": \"幸福\"\n}\n```\n\nApp Search:\n\n```json\n{\n \"id\": \"12345\",\n \"hello\": \"你好\",\n \"love\": \"爱\",\n \"happiness\": \"幸福\"\n}\n```\n\n### Monitoring\n\nAs a best practice, you can [monitor the activity](https://firebase.google.com/docs/extensions/manage-installed-extensions#monitor) of your installed extension, including checks on its health, usage, and logs.\n","readmeContent":"# Elastic App Search extension for Firestore\n\nThis extension syncs data from Google's [Cloud Firestore](https://firebase.google.com/products/firestore) to [Elastic App Search](https://www.elastic.co/app-search/).\n\nOut-of-the-box, Cloud Firestore provides no mechanism for full-text search on data. Syncing your Cloud Firestore data to Elastic App Search not only gives you a mechanism for full-text search on your data, it also lets you enjoy App Search's powerful relevance tuning features and search analytics data.\n\n## Install\n\n### From the web\n\nVisit the following link: https://console.firebase.google.com/project/_/extensions/install?ref=elastic/firestore-elastic-app-search@0.4.1\n\n### From source\n\nAfter pulling this project source locally, follow these steps:\n\n```shell\nnpm install -g firebase-tools\nnpm install\nfirebase login\nfirebase ext:install . --project=\n```\n\n## Contributing\n\nPlan to pull this code and run it locally? See [CONTRIBUTING.md](CONTRIBUTING.md).\n","externalServices":[{"name":"Elastic Enterprise Search","pricingUri":"https://www.elastic.co/pricing/"}],"displayName":"Search with Elastic App Search","systemParams":[{"param":"firebaseextensions.v1beta.function/timeoutSeconds","label":"Function timeout seconds","type":"STRING","description":"How long should functions run before timing out, in seconds (0-540)?","validationRegex":"^[1-9][0-9]{0,2}$","validationErrorMessage":"Function timeout should be an integer number of seconds, between 0 and 540","advanced":true},{"param":"firebaseextensions.v1beta.function/vpcConnector","label":"VPC Connector","type":"STRING","description":"The VPC Network Connector that this cloud function can connect to. It can be either the fully-qualified URI, or the short name of the network connector resource. The format of this field is projects/*/locations/*/connectors/*.","validationRegex":"^projects/([^/]+)/locations/([^/]+)/connectors/([^/]+)$","advanced":true},{"param":"firebaseextensions.v1beta.function/vpcConnectorEgressSettings","label":"VPC Connector Egress settings","type":"SELECT","description":"Controls outgoing traffic when a VPC connector is configured","options":[{"value":"VPC_CONNECTOR_EGRESS_SETTINGS_UNSPECIFIED","label":"Unspecified"},{"value":"PRIVATE_RANGES_ONLY","label":"Private ranges only"},{"value":"ALL_TRAFFIC","label":"All traffic"}],"default":"VPC_CONNECTOR_EGRESS_SETTINGS_UNSPECIFIED","advanced":true},{"param":"firebaseextensions.v1beta.function/minInstances","label":"Minimum function instances","type":"STRING","description":"The minimum number of instances of each function to run at once (0-1000)","default":"0","validationRegex":"^[0-9]*$","validationErrorMessage":"Min instances must be a non-negative integer.","advanced":true},{"param":"firebaseextensions.v1beta.function/maxInstances","label":"Maximum function instances","type":"STRING","description":"The maximum number of instances of each function to run at once","validationRegex":"^[0-9]*$","validationErrorMessage":"Max instances must be a non-negative integer.","advanced":true},{"param":"firebaseextensions.v1beta.function/ingressSettings","label":"Function ingress settings","type":"SELECT","description":"Where should functions allow incoming traffic from?","options":[{"value":"ALLOW_ALL","label":"Allow all"},{"value":"ALLOW_INTERNAL_ONLY","label":"Allow internal only"},{"value":"ALLOW_INTERNAL_AND_GCLB","label":"Allow internal and GCLB"}],"advanced":true},{"param":"firebaseextensions.v1beta.function/labels","label":"Function labels","type":"STRING","description":"Do you wish to set any labels on this instance's Cloud Functions? If so, provide up to 55 labels in the format 'key1:value, key2:value'","validationRegex":"^([a-zžà-ÿ][A-Za-zŽžÀ-ÿ0-9_-]{0,62}:[A-Za-zŽžÀ-ÿ0-9_-]{0,63},\\s*)*([a-zžà-ÿ][A-Za-zŽžÀ-ÿ0-9_-]{0,62}:[A-Za-zŽžÀ-ÿ0-9_-]{0,63})$","advanced":true},{"param":"firebaseextensions.v1beta.function/kmsKeyName","label":"KMS key name","type":"STRING","description":"Do you want to use a Customer Managed Encryption Key (CMEK) to encrypt this extension's functions? If you set this, you must also set a Docker repository encrypted by that key. See https://cloud.google.com/functions/docs/securing/cmek for more details.","validationRegex":"^projects/([^/]+)/locations/([^/]+)/keyRings/([^/]+)/cryptoKeys/([^/]+)$","advanced":true},{"param":"firebaseextensions.v1beta.function/dockerRepository","label":"Docker repository","type":"STRING","description":"What Docker repository should be used to store function images? Default repository will be used if not set.","validationRegex":"^projects/([^/]+)/locations/([^/]+)/repositories/([^/]+)$","advanced":true},{"param":"firebaseextensions.v1beta.function/memory","label":"Function memory","type":"SELECT","description":"How much memory should be allocated to each v1 function?","options":[{"value":"128","label":"128MB"},{"value":"256","label":"256MB"},{"value":"512","label":"512MB"},{"value":"1024","label":"1GB"},{"value":"2048","label":"2GB"},{"value":"4096","label":"4GB"},{"value":"8192","label":"8GB"}],"default":"256","advanced":true}]},"fetchTime":"2023-10-17T11:31:24.999715Z","lastOperationName":"projects/not-a-project/operations/71fad467-3c6e-43c9-bccc-63dac47c8646","state":"ACTIVE"}}
[debug] [2023-10-17T11:31:25.629Z] Created new Extension Source %s projects/not-a-project/sources/aae91f2b-6917-4b50-a845-d5b64f75591c
[debug] [2023-10-17T11:31:25.629Z] >>> [apiv2][query] DELETE https://storage.googleapis.com/firebase-ext-eap-uploads/firebase-archive--58488-i3qPvDbbgHO0-.zip [none]
[debug] [2023-10-17T11:31:25.996Z] <<< [apiv2][status] DELETE https://storage.googleapis.com/firebase-ext-eap-uploads/firebase-archive--58488-i3qPvDbbgHO0-.zip 204
[debug] [2023-10-17T11:31:25.996Z] <<< [apiv2][body] DELETE https://storage.googleapis.com/firebase-ext-eap-uploads/firebase-archive--58488-i3qPvDbbgHO0-.zip undefined
[debug] [2023-10-17T11:31:25.996Z] Cleaned up uploaded source archive
[debug] [2023-10-17T11:31:25.998Z] >>> [apiv2][query] GET https://iam.googleapis.com/v1/roles/secretmanager.secretAccessor [none]
[debug] [2023-10-17T11:31:26.000Z] >>> [apiv2][query] GET https://iam.googleapis.com/v1/roles/datastore.user [none]
[debug] [2023-10-17T11:31:26.984Z] <<< [apiv2][status] GET https://iam.googleapis.com/v1/roles/secretmanager.secretAccessor 200
[debug] [2023-10-17T11:31:26.984Z] <<< [apiv2][body] GET https://iam.googleapis.com/v1/roles/secretmanager.secretAccessor {"name":"roles/secretmanager.secretAccessor","title":"Secret Manager Secret Accessor","description":"Allows accessing the payload of secrets.","includedPermissions":["resourcemanager.projects.get","resourcemanager.projects.list","secretmanager.versions.access"],"stage":"GA","etag":"AA=="}
[debug] [2023-10-17T11:31:26.994Z] <<< [apiv2][status] GET https://iam.googleapis.com/v1/roles/datastore.user 200
[debug] [2023-10-17T11:31:26.994Z] <<< [apiv2][body] GET https://iam.googleapis.com/v1/roles/datastore.user {"name":"roles/datastore.user","title":"Cloud Datastore User","description":"Provides read/write access to data in a Cloud Datastore database. Intended for application developers and service accounts.","includedPermissions":["appengine.applications.get","datastore.databases.get","datastore.databases.getMetadata","datastore.databases.list","datastore.entities.allocateIds","datastore.entities.create","datastore.entities.delete","datastore.entities.get","datastore.entities.list","datastore.entities.update","datastore.indexes.list","datastore.namespaces.get","datastore.namespaces.list","datastore.statistics.get","datastore.statistics.list","resourcemanager.projects.get","resourcemanager.projects.list"],"stage":"GA","etag":"AA=="}
[info]
Extension: Search with Elastic App Search
Description: Syncs documents from a Firestore collection to Elastic App Search to enable full-text search.
Version: 0.4.1
License: Apache-2.0
Resources created:
- shipToElastic (Cloud Function (1st gen)): Function triggered on Create, Update, or Delete of a document in the specified collection which syncs the change to App Search.
- APP_SEARCH_API_KEY (Cloud Secret Manager secret)
External services used:
- Elastic Enterprise Search (https://www.elastic.co/pricing/)
Roles granted:
- Secret Manager Secret Accessor: Allows accessing the payload of secrets.
- Cloud Datastore User: Provides read/write access to data in a Cloud Datastore database. Intended for application developers and service accounts.
[info] i extensions: ensuring required API secretmanager.googleapis.com is enabled...
[debug] [2023-10-17T11:31:28.848Z] >>> [apiv2][query] GET https://serviceusage.googleapis.com/v1/projects/not-a-project/services/secretmanager.googleapis.com [none]
[debug] [2023-10-17T11:31:28.848Z] >>> [apiv2][(partial)header] GET https://serviceusage.googleapis.com/v1/projects/not-a-project/services/secretmanager.googleapis.com x-goog-quota-user=projects/not-a-project
[debug] [2023-10-17T11:31:29.751Z] <<< [apiv2][status] GET https://serviceusage.googleapis.com/v1/projects/not-a-project/services/secretmanager.googleapis.com 200
[debug] [2023-10-17T11:31:29.751Z] <<< [apiv2][body] GET https://serviceusage.googleapis.com/v1/projects/not-a-project/services/secretmanager.googleapis.com [omitted]
[info] ✔ extensions: required API secretmanager.googleapis.com is enabled
[debug] [2023-10-17T11:31:29.754Z] >>> [apiv2][query] GET https://firebase.googleapis.com/v1beta1/projects/not-a-project/adminSdkConfig [none]
[debug] [2023-10-17T11:31:29.993Z] <<< [apiv2][status] GET https://firebase.googleapis.com/v1beta1/projects/not-a-project/adminSdkConfig 200
[debug] [2023-10-17T11:31:29.993Z] <<< [apiv2][body] GET https://firebase.googleapis.com/v1beta1/projects/not-a-project/adminSdkConfig {"projectId":"not-a-project","storageBucket":"not-a-project.appspot.com","locationId":"europe-west"}
[debug] [2023-10-17T11:31:29.994Z] >>> [apiv2][query] GET https://firebase.googleapis.com/v1beta1/projects/not-a-project [none]
[debug] [2023-10-17T11:31:30.256Z] <<< [apiv2][status] GET https://firebase.googleapis.com/v1beta1/projects/not-a-project 200
[debug] [2023-10-17T11:31:30.257Z] <<< [apiv2][body] GET https://firebase.googleapis.com/v1beta1/projects/not-a-project {"projectId":"not-a-project","projectNumber":"1234567890","displayName":"not-a-project","name":"projects/not-a-project","resources":{"hostingSite":"not-a-project-46a13","storageBucket":"not-a-project.appspot.com","locationId":"europe-west"},"state":"ACTIVE","etag":"1_51ebbee2-5413-4433-9d20-5b32e6afd29e"}
[info] i extensions: answer the questions below to configure your extension:
[info]
Cloud Functions location: Choose where you want to deploy the functions created for this extension. For help selecting a location, refer to the location selection guide (https://firebase.google.com/docs/functions/locations).
[info]
Collection path: The path to the collection that you want to sync to App Search.
[info]
Elastic App Search engine name: The name of the Elastic App Search "engine" you want to sync to your collection.
[info]
Elastic App Search private API key: A "private" API key from your Elastic App Search deployment with access to the engine named above. API keys can be found in the App Search "Credentials" page (https://www.elastic.co/guide/en/app-search/current/authentication.html#authentication-api-keys).
[debug] [2023-10-17T11:31:37.362Z] >>> [apiv2][query] GET https://secretmanager.googleapis.com/v1/projects/not-a-project/secrets/ext-firestore-elastic-app-search-APP_SEARCH_API_KEY [none]
[debug] [2023-10-17T11:31:37.593Z] <<< [apiv2][status] GET https://secretmanager.googleapis.com/v1/projects/not-a-project/secrets/ext-firestore-elastic-app-search-APP_SEARCH_API_KEY 404
[debug] [2023-10-17T11:31:37.593Z] <<< [apiv2][body] GET https://secretmanager.googleapis.com/v1/projects/not-a-project/secrets/ext-firestore-elastic-app-search-APP_SEARCH_API_KEY {"error":{"code":404,"message":"Secret [projects/1234567890/secrets/ext-firestore-elastic-app-search-APP_SEARCH_API_KEY] not found.","status":"NOT_FOUND"}}
[debug] [2023-10-17T11:31:39.478Z] >>> [apiv2][query] POST https://secretmanager.googleapis.com/v1/projects/not-a-project/secrets secretId=ext-firestore-elastic-app-search-APP_SEARCH_API_KEY
[debug] [2023-10-17T11:31:39.479Z] >>> [apiv2][body] POST https://secretmanager.googleapis.com/v1/projects/not-a-project/secrets {"name":"ext-firestore-elastic-app-search-APP_SEARCH_API_KEY","replication":{"automatic":{}},"labels":{"firebase-extensions-managed":"firestore-elastic-app-search"}}
[debug] [2023-10-17T11:31:39.558Z] <<< [apiv2][status] POST https://secretmanager.googleapis.com/v1/projects/not-a-project/secrets 400
[debug] [2023-10-17T11:31:39.558Z] <<< [apiv2][body] POST https://secretmanager.googleapis.com/v1/projects/not-a-project/secrets {"error":{"code":400,"message":"Constraint constraints/gcp.resourceLocations violated for [orgpolicy:projects/1234567890] attempting to create a secret in [global]. For more information, see https://cloud.google.com/resource-manager/docs/organization-policy/defining-locations.","status":"FAILED_PRECONDITION","details":[{"@type":"type.googleapis.com/google.rpc.PreconditionFailure","violations":[{"type":"constraints/gcp.resourceLocations","subject":"orgpolicy:projects/1234567890","description":"Constraint constraints/gcp.resourceLocations violated for [orgpolicy:projects/1234567890] attempting to create a secret in [global]. For more information, see https://cloud.google.com/resource-manager/docs/organization-policy/defining-locations."}]}]}}
[error]
[error] Error: HTTP Error: 400, Constraint constraints/gcp.resourceLocations violated for [orgpolicy:projects/1234567890] attempting to create a secret in [global]. For more information, see https://cloud.google.com/resource-manager/docs/organization-policy/defining-locations.
[debug] [2023-10-17T11:31:39.644Z] Error Context: {
"body": {
"error": {
"code": 400,
"message": "Constraint constraints/gcp.resourceLocations violated for [orgpolicy:projects/1234567890] attempting to create a secret in [global]. For more information, see https://cloud.google.com/resource-manager/docs/organization-policy/defining-locations.",
"status": "FAILED_PRECONDITION",
"details": [
{
"@type": "type.googleapis.com/google.rpc.PreconditionFailure",
"violations": [
{
"type": "constraints/gcp.resourceLocations",
"subject": "orgpolicy:projects/1234567890",
"description": "Constraint constraints/gcp.resourceLocations violated for [orgpolicy:projects/1234567890] attempting to create a secret in [global]. For more information, see https://cloud.google.com/resource-manager/docs/organization-policy/defining-locations."
}
]
}
]
}
},
"response": {
"statusCode": 400
}
}
```
### Alternatives tried:
- ❌ Pre-create the secret `ext-firestore-elastic-app-search-APP_SEARCH_API_KEY` with the label `firebase-extensions-managed: true`, however, that only leads to the extension trying to create another secret with some suffix attached to it.
Contributor guide
Assessment
This issue has not been assessed yet.