Azure / Azure/azure-cli-extensions

[acrcssc] Cannot respond to request for authentication during OCI artifact push - root cause identified, not Cloud Shell-specific

Open
#10,288 1 comment 0 reactions 0 assignees View on GitHub
act-observability-squad Container Registry customer-reported question Service Attention
Dominant language
Python
Stars
454
Forks
1.7k
Avg merge
2d 19h
Merged PRs (30d)
64

Description

### Related issues
- Azure/azure-cli-extensions#10003
- Azure/azure-cli#32817

Those reports attribute this to Azure Cloud Shell's credential handoff. I hit the identical failure on a plain local Linux install (non-Cloud-Shell) and traced it to a concrete bug in how `acrcssc` uses `oras-py`'s `TokenAuth` backend — it does not appear to be environment-specific.

### Environment
```
azure-cli: 2.75.0 (also reproduced on 2.90.0 before downgrading for an unrelated SDK issue)
acrcssc: 1.0.0b7
oras (vendored): 0.2.25
```

### Repro
```bash
az acr supply-chain workflow create \
-r -g -t continuouspatchv1 \
--config ./continuouspatching.json --schedule 1d --run-immediately
```

Fails with:
```
Failed to push OCI artifact to ACR: Cannot respond to request for authentication.
```

`--dry-run` succeeds (it never pushes), confirming the bug is isolated to the OCI artifact push step.

### Root cause

`_ociartifactoperations.py::_get_acr_token()` shells out to `az acr login --expose-token`, which returns an ACR **refresh token** (explicitly *not* a usable access token — `az` itself warns about this). `_oras_client()` then does:

```python
token = _get_acr_token(registry.name, subscription)
client = OrasClient(hostname=str.lower(registry.login_server), auth_backend="token")
client.login(BEARER_TOKEN_USERNAME, token)
```

But `oras.provider.Registry.login()` only ever calls `self.auth.set_basic_auth(username, password)` — it **never** calls `set_token_auth()`. So `TokenAuth.token` stays `None` after login.

Separately, `oras.provider.Registry.do_request()` contains:

```python
if headers is not None and isinstance(self.auth, oras.auth.TokenAuth):
headers.update(self.auth.get_auth_header())
```

`TokenAuth.get_auth_header()` unconditionally returns `{"Authorization": "Bearer %s" % self.token}`. Since `self.token is None`, this attaches a literal `Authorization: Bearer None` header on the **first** request, before any real 401 challenge/response cycle happens.

ACR sees a malformed-but-present Bearer credential and responds `403` **without** a `WWW-Authenticate` header (that's only sent on clean, unauthenticated 401s). `TokenAuth.authenticate_request()` then can't recover:

```python
authHeaderRaw = original.headers.get("Www-Authenticate")
if not authHeaderRaw:
logger.debug("Www-Authenticate not found in original response, cannot authenticate.")
return headers, False # -> do_request() raises ValueError("Cannot respond to request for authentication.")
```

I confirmed this by replaying the exact same "raw refresh token as Bearer" request manually — ACR returns `401 insufficient_scope` *with* `WWW-Authenticate` in isolation, but the premature/poisoned first request in the real flow behaves differently and the extension never gets a chance to recover.

I also confirmed the `oras-py` "basic" backend (`auth_backend="basic"`) is **not** a valid alternative — its `authenticate_request()` just resends raw HTTP Basic auth to the blob endpoint, which ACR's data-plane API doesn't accept; no real OAuth2 exchange happens there either.

### Fix

Rather than relying on `oras`'s broken token negotiation, perform the ACR `refresh_token` → scoped `access_token` exchange explicitly, and hand the resulting token directly to the auth backend via `set_token_auth()` (bypassing `login()`'s broken basic-auth-only path entirely):

**Before** (`azext_acrcssc/helper/_ociartifactoperations.py`):
```python
def _oras_client(registry):
resourceid = parse_resource_id(registry.id)
subscription = resourceid[SUBSCRIPTION]

try:
token = _get_acr_token(registry.name, subscription)
client = OrasClient(hostname=str.lower(registry.login_server), auth_backend="token")
client.login(BEARER_TOKEN_USERNAME, token)
logger.debug(f"Login to ACR {registry.name} completed successfully.")
except Exception as exception:
raise AzCLIError(f"Failed to login to Artifact Store ACR {registry.name}: {exception}")

return client
```

**After:**
```python
import requests

def _oras_client(registry):
resourceid = parse_resource_id(registry.id)
subscription = resourceid[SUBSCRIPTION]

try:
refresh_token = _get_acr_token(registry.name, subscription)
login_server = str.lower(registry.login_server)
# oras-py's TokenAuth backend does not perform the ACR
# refresh_token -> access_token exchange correctly (it sends a
# premature "Bearer None" header on the first request, and cannot
# recover afterwards). Perform the exchange explicitly instead.
scope = f"repository:{CSSC_WORKFLOW_POLICY_REPOSITORY}/{CONTINUOUSPATCH_OCI_ARTIFACT_CONFIG}:pull,push"
exchange_resp = requests.post(
f"https://{login_server}/oauth2/token",
data={
"grant_type": "refresh_token",
"service": login_server,
"scope": scope,
"refresh_token": refresh_token,
},
timeout=30,
)
exchange_resp.raise_for_status()
access_token = exchange_resp.json()["access_token"]

client = OrasClient(hostname=login_server, auth_backend="token")
client.auth.set_token_auth(access_token)
logger.debug(f"Login to ACR {registry.name} completed successfully.")
except Exception as exception:
raise AzCLIError(f"Failed to login to Artifact Store ACR {registry.name}: {exception}")

return client
```

I applied this locally against `~/.azure/cliextensions/acrcssc/` and confirmed `az acr supply-chain workflow create ... --run-immediately` now completes successfully end-to-end (artifact push, ARM task deployment, and workflow trigger all succeed).

### Suggested longer-term fix
Ideally this gets fixed upstream in `oras-py` itself (`TokenAuth`/`do_request` shouldn't attach a `Bearer None` header when no token has been obtained yet), but the workaround above avoids depending on a fix there and is a minimal, self-contained change to `acrcssc`.

Happy to open a PR with this change if useful.

Contributor guide

Open the contributing guide

Research direction

Start in azext_acrcssc/helper/_ociartifactoperations.py, especially _get_acr_token() and _oras_client(), then reproduce with the supplied az acr supply-chain workflow create command and --run-immediately. Review the oras-py TokenAuth behavior and the ACR /oauth2/token exchange described in the issue. Done means the command completes the artifact push, ARM task deployment, and workflow trigger without the authentication error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, cli
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.