[Bug]: Gateway revokes every locally-held API key on restart when the control plane does not own them

Please select the area the issue is related to

Gateway

Please select the aspect the issue is related to

Aspect/API (API backends, definitions, contracts, interfaces, OpenAPI)

Description

On every control-plane (re)connect the gateway-controller runs three bulk reconcilers: subscription plans, subscriptions, and API keys (client.go#L527 on first connect, #L554 on reconnect). All three treat the control plane as authoritative: fetch the CP's set, upsert it, then delete every local row the CP did not return.

Plans and subscriptions opt out of this when the CP is an on-prem APIM:

• syncSubscriptionPlans — client.go#L850
• syncSubscriptionsForExistingAPIs — client.go#L926

syncAPIKeysForExistingArtifacts (client.go#L1048) has no equivalent guard. It only narrows which kinds it syncs (onPremSupportedAPIKeyKinds, client.go#L1038 and the skip at #L1097) and then reconciles RestApi keys for real.

Because the gateway never pushes API keys up to any control plane (the only bottom-up call is /artifacts/import-gateway-artifacts, itself !IsOnPrem()-gated), any key that originated on the gateway is permanently invisible to the CP. The reconciliation therefore classifies it as stale and deletes it:

artifactUUIDs := artifactUUIDsByKind[kind]
staleKeys, err := c.db.ListAPIKeysForArtifactsNotIn(artifactUUIDs, fetchedUUIDs)
...
if err := c.db.DeleteAPIKeysByUUIDs(staleUUIDs); err != nil {
...
    for _, k := range staleKeys {
        c.apiKeyService.PublishAPIKeyEvent("DELETE", k.ArtifactUUID, k.UUID, ...)
    }

client.go#L1166-L1186

When the CP returns an empty set, ListAPIKeysForArtifactsNotIn degenerates to SELECT ... WHERE gateway_id = ? AND artifact_uuid IN (<every known REST API>) — every key on every API. Each deletion publishes a DELETE event, which handleAPIKeyRevoke turns into a policy-engine revocation, so the key stops authenticating immediately.

Nothing in the reconciliation distinguishes a key's origin. The source column ("local" / "external") is not consulted, and in any case records how the key value arrived, not who owns the record: a plaintext key injected over the management API and a key pushed by platform-API are both stamped "external" (api_key.go#L995, #L1008).

Impact by deployment

Subscriptions and plans are unaffected in both deployments. Note the reconciliation keys solely off "is this UUID in the set the CP's backfill just returned" — neither the injection transport (the WebSocket apikey.created channel at client.go#L1440, which has no on-prem branch, or the gateway management API) nor the source value affects it. On-prem, the practical result observed is that every API key on the gateway is revoked on every restart, regardless of how it got there, because APIM's backfill returns none of them.

There is no recovery path through the event channel: the injection events succeeded, and the revocation happens afterwards on the gateway side, so an event mediator's failed-event queue stays empty and a requeue is a no-op. The only recovery is re-issuing every key.

Additional notes

• If the CP call fails (non-200 / transport error), FetchAPIKeysByKind errors and the loop continues before the delete — a CP that is down is harmless; only a CP that answers with a set missing these keys triggers the loss.
• issuer is passed to the CP as a server-side filter (api_utils.go#L329), from systemConfig.APIKey.Issuer. A mismatch between the gateway's configured issuer and the issuer on the CP's keys empties the fetched set and triggers the same wipe — in either deployment.
• Keys the CP returns as non-active, or without a sha256 hash, are skipped by the fetch and so also counted as stale.
• There is no test covering the on-prem branch of this sync; existing coverage only exercises the platform-API path (pkg/controlplane/api_deleted_test.go).

Steps to Reproduce

On-prem APIM as control plane (primary case):

1. Start a gateway attached to an on-prem APIM control plane, with at least one deployed REST API.
2. Issue an API key for that API — directly via the gateway management API (POST /apis/{id}/api-keys).
3. Confirm GET /rest-apis/{api}/api-keys lists the key and that a request carrying it returns 200.
4. Restart the controller: docker compose restart gateway-controller.
5. Observe in the controller logs:

   msg="Starting API key sync at gateway startup"
   msg="Processing API key revoke event" api_id=...
   msg="Revoking API key" api_id=... api_key=<key-name>
   

6. The same request now returns 401 and GET /rest-apis/{api}/api-keys returns 0 keys. Subscriptions on the same API are still intact.

platform-API as control plane (secondary case):

1. Attach a gateway to a platform-API control plane.
2. Create a key directly on the gateway via POST /apis/{id}/api-keys (not in the control plane).
3. Confirm it authenticates.
4. Restart the controller — the key is deleted and revoked by the same reconciliation. A key created in the control plane and pushed down survives.

Severity Level of the Issue

Severity/Critical (Core functionality is broken but there is a workaround. Need urgent attention)

Environment Details (with versions)

No response

Start in gateway/gateway-controller/pkg/controlplane/client.go at syncAPIKeysForExistingArtifacts and compare its behavior with syncSubscriptionPlans and syncSubscriptionsForExistingAPIs. Review pkg/controlplane/api_deleted_test.go and the storage queries around ListAPIKeysForArtifactsNotIn, then add coverage for the on-prem and locally-held key cases. Done means locally-held keys are not deleted during reconciliation while control-plane-owned keys continue to sync correctly.