hf-gce: GCP authentication fails with AttributeError ('Request' has no 'session') in published RPM
- Dominant language
- Python
- Stars
- 7
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
## Expected Behavior
The GCE provider (`hf-gce`) should successfully authenticate to GCP and call the Compute API to create or delete VMs in the configured Instance Group when `requestMachines` or `requestReturnMachines` is invoked.
## Actual Behavior
The provider catches a `google-auth` exception during authentication and exits the command without making any GCP API call. No VM is created or deleted in the configured Instance Group.
This affects every command that requires GCP authentication (`requestMachines`, `requestReturnMachines`).
- From the **CLI**, the wrapper script returns the error text on stdout and exits:
```
Error: 'Request' object has no attribute 'session'
```
- From the **Management Console**, the schedule appears as `Closed` shortly after creation and no machine is provisioned. The failure is visible only in the provider log.
This affects the GCE provider (`hf-gce`) only. GKE (`hf-gke`) authenticates via kubeconfig and isn't affected.
## Root Cause
This is an upstream bug in the `google-auth` Python library, introduced when mTLS support was added to metadata server connections in `google-auth 2.44.0` and fixed in `google-auth 2.49.1`.
Reference: https://github.com/googleapis/google-cloud-python/issues/16035
The `pyproject.toml` in this repo declares `google-auth>=2.40.3` with no upper bound, and the CI workflow uses `uv pip install -r pyproject.toml`, which resolves to whatever is the latest matching version at build time. The currently published RPM was built at a point in time when the resolver picked a version in the buggy range (`2.44.0`–`2.48.x`).
## Steps to Reproduce the Problem
> **Note:** This reproduction applies `2>/dev/null` to the wrapper scripts. Without it, library warnings printed on stderr are mixed into the script output and prevent HostFactory from parsing the response, masking the underlying authentication error described here. The stderr behavior itself is a separate issue (see #86). The `2>/dev/null` here is purely a reproduction aid to isolate this bug.
### Management Console
1. Install the GCE provider from the published RPM:
```bash
sudo tee /etc/yum.repos.d/google-cloud-symphony-connector.repo << EOM
[google-cloud-symphony-connector]
name=Google Cloud Symphony Connector
baseurl=https://packages.cloud.google.com/yum/repos/google-cloud-symphony-connector-x86-64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOM
yum install hf-gcpgce-provider.x86_64
```
2. Configure the provider instance following the standard [install documentation](https://github.com/google/symphony-gcp/blob/main/hf-provider/SYMPHONY_GCE_INSTALL.md).
3. Apply `2>/dev/null` to the binary call in each wrapper script so the authentication error is not obscured by library warnings:
```bash
sed -i 's|hf-gce \(.*\) -f \$inJson$|hf-gce \1 -f $inJson 2>/dev/null|' \
$HF_TOP/1.2/providerplugins/gcpgce/scripts/*.sh
```
4. From the Symphony Management Console, request a new cloud/machine request with the GCE provider.
5. Observe that the request closes within a few seconds with status `Closed`.
6. No VM is created in GCP.
7. The provider log at `$HF_TOP/log/gcpgceinst-provider.*.log` shows the authentication error:
```diff
INFO - DISPATCHING|command: requestMachines; payload: ...
INFO - cmd_request_machines {'template': {'templateId': 'template-gcp-01', 'machineCount': 1}}
INFO - Performing process-level initialization for GCE provider.
INFO - Database initialization complete.
INFO - Initialization complete.
INFO - request: template=Template(...) gcp_zone='us-central1-c' gcp_instance_group='aa-instance-group-1'
INFO - Received request to provision 1 machines with prefix sym-
- ERROR - Error creating compute.CreateInstancesInstanceGroupManagerRequest: 'Request' object has no attribute 'session'
```
### CLI
1. Trigger a `requestMachines` call directly:
```bash
echo '{"template": {"templateId": "template-gcp-01", "machineCount": 1}}' > /tmp/req.json
HF_PROVIDER_CONFDIR=$HF_TOP/conf/providers/gcpgceinst HF_DBDIR=$HF_TOP/db $HF_TOP/1.2/providerplugins/gcpgce/scripts/requestMachines.sh -f /tmp/req.json
```
2. Observe the error returned on stdout:
```diff
- Error: 'Request' object has no attribute 'session'
```
3. Confirm in GCP that no VM was created in the configured Instance Group.
## Environment
- Provider affected: `hf-gce`
- RPM build: `hf-gcpgce-provider-1.0.0-110.el10.x86_64`
- IBM Spectrum Symphony: 7.3.2 (Fix 602717)
- HostFactory: 1.2
- OS: Rocky Linux 8/9/10, RHEL 8/9/10
Contributor guide
Research direction
Start with pyproject.toml's google-auth constraint and the CI workflow's uv pip install step, then trace hf-gce authentication used by requestMachines and requestReturnMachines. Reproduce with the published RPM and wrapper commands in the issue; done means authentication succeeds and Compute API calls provision or return VMs without the AttributeError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- gcp, python
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100