googleapis / googleapis/google-cloud-node

Consider option to bypass Metadata-Flavor response header check

Open
#7,688 3 comments 3 reactions 0 assignees View on GitHub
library: google-cloud-node-core type: feature request
Dominant language
TypeScript
Stars
3.2k
Forks
712
Avg merge
2d 3h
Merged PRs (30d)
99

Description

### What would you like to see in the library?

My team recently experienced a problem in one of our applications using the `@google-cloud/secret-manager` package where secret lookups starting failing with the following error (originating from this library):

```
/var/opt/runbot/api/node_modules/@google-cloud/secret-manager/node_modules/gcp-metadata/build/src/index.js:121
throw new Error(`Invalid response from metadata service: incorrect ${exports.HEADER_NAME} header.`);
^
Error: Invalid response from metadata service: incorrect Metadata-Flavor header.
at metadataAccessor (/var/opt/runbot/api/node_modules/@google-cloud/secret-manager/node_modules/gcp-metadata/build/src/index.js:121:19)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async GoogleAuth._GoogleAuth_getUniverseFromMetadataServer (/var/opt/runbot/api/node_modules/@google-cloud/secret-manager/node_modules/google-auth-library/build/src/auth/googleauth.js:791:26)
at async GoogleAuth.getUniverseDomain (/var/opt/runbot/api/node_modules/@google-cloud/secret-manager/node_modules/google-auth-library/build/src/auth/googleauth.js:190:168)
at async GoogleAuth.getApplicationDefaultAsync (/var/opt/runbot/api/node_modules/@google-cloud/secret-manager/node_modules/google-auth-library/build/src/auth/googleauth.js:256:42)
at async GoogleAuth.getClient (/var/opt/runbot/api/node_modules/@google-cloud/secret-manager/node_modules/google-auth-library/build/src/auth/googleauth.js:678:17)
at async GrpcClient._getCredentials (/var/opt/runbot/api/node_modules/@google-cloud/secret-manager/node_modules/google-gax/build/src/grpc.js:145:24)
at async GrpcClient.createStub (/var/opt/runbot/api/node_modules/@google-cloud/secret-manager/node_modules/google-gax/build/src/grpc.js:308:23)
```

Upon investigation, we discovered the following:

1. Our application was using version 5.0.1 of `@google-cloud/secret-manager`, which has a transitive dependency on `google-auth-library` (via the `google-gax` dependency) with a semver range of `^9.0.0`.
2. The `google-auth-library` [changed in version 9.10.0](https://github.com/googleapis/google-auth-library-nodejs/compare/v9.9.0...v9.10.0#diff-535a540500eec341838bd88c84b1795313a8ff9b556bad87ef89fb2fca061630) from requesting a `universe_domain` metadata endpoint (underscore) to a `universe-domain` endpoint (dash). See changes to the _src/auth/googleauth.ts_ module.
3. The GCP metadata service for our GCE hosts get different responses for the `universe_domain` (legacy) endpoint in different zones. We had one GCE instance in zone `europe-west1-b` receiving a 404 HTML response **WITH** the necessary `Metadata-Flavor: Google` response header, and one GCE instance in `europe-west1-c` receiving a 200 text response **WITHOUT** the necessary `Metadata-Flavor: Google` response header. Why this is the case, I have no idea.

The proper fix is to upgrade our `google-auth-library` transitive dependency (within our package-lock.json tree) to a version >= 9.10.0, but attempting to do this by upgrading our direct dependency on `@google-cloud/secret-manager` to its latest version (5.6.0 as of this writing) **DID NOT** change the versions of the transitive dependencies `google-auth-library` or `google-gax` in our package-lock.json tree, supposedly because the declared semver range for their dependencies ([`@google-cloud/secret-manager@5.6.0`](https://www.npmjs.com/package/@google-cloud/secret-manager/v/5.6.0?activeTab=code) declares `"google-gax": "^4.0.3"` and [`google-gax@4.4.1`](https://www.npmjs.com/package/google-gax/v/4.4.1?activeTab=code) declares `"google-auth-library": "^9.3.0"`) was still satisfied by the versions already included in our tree. This is a bit frustrating but is easily worked around by doing `npm i google-auth-library@^9.0.0` and then `npm uninstall google-auth-library`.

However, due to the mysterious nature of the GCP metadata service and its unknown release/update cycle and apparent potential for breaking changes between different zones within the same GCP region at the same time, it would have been quicker and easier for us to resolve our broken application if there was some way to tell this library (`gcp-metadata`) to log a warning when the metadata service response is missing an expected header rather than have it always throw an error.

My proposal is to introduce an environment variable (e.g. `GCE_METADATA_ALLOW_MISSING_HEADER=true`) that would allow for this behavior.

It should obviously only be needed/utilized in rare scenarios, but since there seems to be inconsistency between the metadata service on different hosts in different zones, this would allow for a safety hatch if an unannounced breaking change to the metadata service is deployed.

Note that when I made this change manually to the installed version of this library on our application host, everything else worked as expected, including successful secret lookups from the Secret Manager.

### Describe alternatives you've considered

_No response_

### Additional context/notes

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.