DRAFT: OpenID4VCI delivery: classify errors, let the caller choose delivery mode, make delivery status visible

Status: first draft. Written up from the review of #4471 to have something concrete to discuss. Nothing in here is decided. Expect it to change after discussion.

Related: #4469, #4470, #4471

Context

#4469 decided to add a retry queue with a dead-letter queue (DLQ) for the OpenID4VCI credential offer, gate the gRPC fallback behind a 24 hour retry window, and backport that as a patch to v5.4 and v6.2. #4471 implements the queue. Reviewing #4471 raised questions that the retry queue alone does not answer. This issue writes down those questions and a proposal, so we can decide before #4471 merges rather than fix it afterwards.

The short version: the queue retries every failure the same way, the caller cannot see that a credential was queued, and the default behaviour changes for every existing integration. The care workflow case makes the second point urgent. When a call center grants night care access and the credential silently enters a retry queue, nobody at the desk knows that access is not going to work.

Problems with #4471 as it stands

• Every error is retried for 24 hours: a credential without a valid subject DID, a local store failure, a wallet that returns HTTP 400, and a wallet that is restarting. Some can never succeed. Others need a person to fix configuration, and nobody is told.
• The HTTP status code is lost. httpDo in vcr/openid4vci/issuer_client.go turns a non-2xx response into a plain string error. No caller can distinguish 404 from 503.
• Queueing is invisible to the caller. Issue() returns the credential with a nil error whether it was delivered or queued. The API returns 200 either way.
• The default changes for every existing integration. Today a misconfigured counterpart still gets the credential immediately over the gRPC fallback. With #4471 the same call silently takes up to 24 hours. Because the fallback hides misconfiguration, those counterparts exist and do not get fixed.
• A store failure after a successful offer is treated as a delivery failure. The next attempt sends the offer again. The receiver's store deduplicates identical credentials, so this is harmless today, but it makes a local problem look like a delivery problem.
• One goroutine per job, each with its own backoff. A setup run issuing hundreds of credentials to a wallet that is down produces hundreds of loops against the same endpoint.
• Give-up means gRPC publish, which is the behaviour #4469 wants to move away from.
• The job holds the full signed credential in a KV shelf. Dead-lettered jobs are kept forever.

Two kinds of issuance

Issuance happens in two situations with different expectations.

In a care workflow, an application issues an authorization credential because someone needs access now. If delivery fails, the person at the desk must know immediately so they can arrange an alternative.

In a setup or onboarding flow, an administrator issues organisation or identity credentials. A delivery that completes later is fine, as long as its status can be checked.

The credential type is a proxy for this distinction, not a reliable one. Authorization credentials are also issued in bulk during setup. The application calling the issue API knows which situation applies. The node does not. The design lets the caller choose.

On "checking underlying errors is complex and error prone"

That objection was raised on #4469 and it deserves a direct answer. The classifier proposed below is two checks: the HTTP status code range, and whether the transport error is a timeout or a dial error. Everything it cannot place goes to the transient bucket, where the retry window handles it. The classifier only removes cases that are certain: local errors, and responses where the counterpart told us it rejects the request. #4470 already classifies one such case (ErrIdentifierNotConfigured, local misconfiguration, no retry). This proposal extends the same idea to the other certain cases.

The cost of not classifying is the incident described above: a caller who waits a day for a failure that was known at the first attempt.

Proposal

1. Classify delivery errors

Return core.HttpError from httpDo for non-2xx responses. The type already exists and carries the status code and truncated body. Transport errors keep their wrapped net.Error.

Wrap only the offer step in issueUsingOpenID4VCI in a typed delivery error. Everything before it (subject DID, issuer identifier discovery, session store) is a local error and fails the call. Technical errors while resolving the wallet's DID count as delivery errors, since they concern the counterpart. Functional resolve errors (DID not found, deactivated) keep their current meaning: the wallet does not support OpenID4VCI.

Two classes of delivery error:

• Transient. The counterpart may recover on its own: connection refused or reset, timeouts, temporary DNS failures, HTTP 408, 429, and 5xx.
• Permanent. A person has to change something: any other 4xx (including 404 on the metadata URL), TLS certificate errors, DNS name not found, metadata without a credential offer endpoint, an offer response with an unexpected status, JSON decoding errors.

Connection refused is deliberately transient. It is what a node in a maintenance window looks like, and also what a decommissioned wallet looks like. A single response cannot tell those apart. The retry window resolves that ambiguity, not the classifier.

A store failure after a successful offer is not a delivery error. The wallet has the offer. Log it and report success, or retry only the store write. Never re-send the offer.

2. Delivery mode on the issue request

Add a field to the issue request next to publishToNetwork and visibility, with two values:

• immediate (default): try OpenID4VCI once. On failure, publish over the gRPC network if a publisher is configured, otherwise return an error. Never queue. This is what master does today.
• background: on a transient error, queue the delivery and return 202. On a permanent error, behave as immediate.

In immediate mode the error response carries the error class (unreachable or rejected) and the underlying reason in the problem details body. Those lead to different phone calls.

Defaulting to immediate keeps the change additive. No existing integration gets a surprise delay. Integrators opt into background once their application can show a pending state.

3. Delivery status visible in the response

The issue endpoint returns the bare credential as its body, so a status field cannot go there without a schema change. Use the status code and a response header instead: 200 with X-Nuts-Delivery: delivered or published, 202 with X-Nuts-Delivery: queued and a Location header pointing at the status resource. Header name to be decided.

Add a status resource that returns the delivery state of a credential by ID: pending, delivered, published, or failed, with attempt count, last error class and message, and next attempt time. Success must be a persisted terminal state, not the deletion of the job. Otherwise "delivered" and "never queued" are indistinguishable afterwards. Terminal records get a retention period. This subsumes the diagnostics and requeue endpoint planned in #4469.

4. Idempotent issuance

Issuing the same credential twice, because the client retried or a user clicked twice, must not produce two signed credentials. Accept an idempotency key on the issue request, scoped to the issuer DID. The node stores key to credential ID and returns the existing credential and its delivery status on a repeat. A content hash of the request parameters is a possible fallback for clients that send no key, but it cannot distinguish a retry from a deliberate re-issuance and needs a short TTL.

5. Queue design

• Group jobs by wallet identifier. Probe a wallet once per backoff step and drain its jobs in order when it answers. A wallet known to be down also lets immediate mode fail at once instead of waiting for a timeout on every call.
• Give up at the earlier of the retry window and the credential's expiration date.
• Give-up records a failure. Publishing over gRPC after give-up is a separate, opt-in setting, so that permanent errors in background mode do not turn into instant gRPC publishing.
• Store jobs in SQL, next to the issuer store. The status resource needs lookup by credential ID, wallet, and state. SQL also allows writing the credential and its job in one transaction, so a crash between the two writes cannot leave a credential nobody will deliver.
• Store the credential in the issuer store only once delivery has succeeded, or remove it on final failure. A failed immediate call must not leave a signed credential behind that nobody holds.
• Bound dead-letter retention. Jobs hold the full signed credential, and identity credentials carry personal data.
• Reuse or delete the session store flow and pre-authorized code on each attempt instead of creating a new one per attempt.

6. Observability

Expose queue depth, dead-letter count, and retries per wallet as Prometheus metrics, and show queue depth in the diagnostics endpoint. Log a retry at Debug and a give-up at Warn. A callback on terminal states would let applications avoid polling, but the node has no such facility and it is out of scope here.

Sequencing and backports

#4469 plans a patch backport of the retry queue to v5.4 and v6.2. Most of this proposal is not patch material. Split it:

1. Patch-safe, can ship with or before #4471: the typed HTTP error, the error classifier, permanent errors falling back (or failing) at once, and the 200/202 plus header signal. Improves the current fallback path on master even without a queue, because the fallback log then states why OpenID4VCI failed.
2. Minor: delivery mode field with immediate as default, classified error response, idempotency key, status resource.
3. Minor: per-wallet grouping, SQL job store, store-after-delivery, metrics.

If the queue ships in a patch before step 2, it must be opt-in by configuration, so a patch does not change delivery timing for existing callers.

Open questions

• #4469 gates the gRPC fallback behind 24 hours because a DAG publish is irreversible and network-wide. A later comment on the same issue says immediate fallback is harmless because the receiver deduplicates. This proposal takes the second position for immediate mode, since that is master's behaviour today. Confirm.
• Should immediate mode with no gRPC publisher revoke the credential it signed, or rely on store-after-delivery from step 3?
• Idempotency key as a header (IETF Idempotency-Key draft) or a request body field?
• Header name and exact values for the delivery status signal.

This is a first-draft design spanning vcr/openid4vci/issuer_client.go, httpDo, issueUsingOpenID4VCI, the issue endpoint, queue storage, and diagnostics. Start by reviewing #4471 and the named entry points, then resolve the open questions before choosing an implementation scope. Done is not yet defined because the proposal and sequencing remain undecided.