apache / apache/apisix-ingress-controller

bug: standalone/ADC - an out-of-band `secrets` resource deadlocks the sync loop (`secrets_conf_version` 400), blocking all reconciliation

Open
#2,821 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
1.1k
Forks
390
Avg merge
3d 7h
Merged PRs (30d)
12

Description

### Current Behavior

We run APISIX in **API-driven standalone** mode driven by the ingress controller
(ADC, `provider.type: apisix-standalone`). We wanted a route's plugin to reference a
Vault secret via APISIX's native `$secret://vault/...`. There is no CRD or
`GatewayProxy` field we could find to declare the backing `secrets` secret-manager, so
the only way to get one onto the data plane is to register it directly on APISIX's
standalone Admin API (out of band).

Doing that **deadlocks the controller**. Once the out-of-band manager bumps APISIX's
`secrets_conf_version` from `0` to `1`, every subsequent controller/ADC full-config
push is rejected:

```
PUT /apisix/admin/configs → 400 Bad Request:
secrets_conf_version must be greater than or equal to (1)
```

ADC has no `secrets` resource type, so its push carries `secrets_conf_version: 0`
while APISIX is now at `1`; APISIX's monotonic version check rejects the **whole**
push. Because each standalone update replaces the full configuration **atomically**,
nothing reconciles after that: the out-of-band manager is **not** wiped, the controller
is simply **stuck**, and even unrelated routes stop applying (they 404 at the gateway).

This looks like a robustness bug independent of the secret use case: once
`secrets_conf_version` is `> 0` by **any** means (an out-of-band manager, a leftover
from a prior setup), ADC - which always sends `0` - can never sync again, and it takes
all other resources down with it.

For reference, APISIX itself handles `$secret://` fine in this mode - we confirmed it
end-to-end with two standalone docker/podman baselines (in the attached bundle), so the
problem is on the controller/ADC side, not APISIX:
- `compose/api-driven/` - in API-driven standalone, APISIX resolves `$secret://vault/...`
when a `secrets` manager is present in the `/configs` payload.
- `compose/file-driven/` - the documented `apisix.yaml` `secrets:` block resolves, but
that mode disables the Admin API the controller drives (so it can't be used together
with the controller).

### Expected Behavior

- The controller/ADC should not deadlock on a resource type it doesn't manage: it
should send a valid (or otherwise tolerated) `secrets_conf_version` so a stray or
out-of-band `secrets` resource can't block reconciliation of everything else.
- Ideally there would be a supported way to declare an APISIX `secrets` secret-manager
through the controller (e.g. a `secrets` field on `GatewayProxy`, or a dedicated
CRD), so `$secret://vault/...` references can be used. Today we can't find one.
(`secretRef` reads a Kubernetes Secret and injects plaintext - a different mechanism,
not APISIX's native `$secret://` manager.)

### Error Logs

Controller `manager` container - this block repeats on **every** sync attempt after the
out-of-band `secrets` resource exists:

```
2026-07-22T09:14:48.222Z INFO provider.client client/client.go:214 syncing all resources
2026-07-22T09:14:48.239Z ERROR provider.executor client/executor.go:420 ADC Server sync failed {"result": {"status":"all_failed","total_resources":0,"success_count":0,"failed_count":1,"success":[],"failed":[],"endpoint_status":[{"server":"http://10.244.0.3:9180","success":false,"reason":"PUT http://10.244.0.3:9180/apisix/admin/configs, responded with status 400 Bad Request, error_msg: secrets_conf_version must be greater than or equal to (1)"}]}, "error": "ADC Server sync failed (standalone mode): http://10.244.0.3:9180: PUT http://10.244.0.3:9180/apisix/admin/configs, responded with status 400 Bad Request, error_msg: secrets_conf_version must be greater than or equal to (1)"}
2026-07-22T09:14:48.240Z ERROR provider.executor client/executor.go:153 failed to run http sync for server {"server": "http://10.244.0.3:9180", "error": "ServerAddr: http://10.244.0.3:9180, Err: http://10.244.0.3:9180: PUT http://10.244.0.3:9180/apisix/admin/configs, responded with status 400 Bad Request, error_msg: secrets_conf_version must be greater than or equal to (1)"}
2026-07-22T09:14:48.240Z ERROR provider.client client/client.go:306 failed to execute adc command {"config": {"name":"GatewayProxy/ingress-apisix/apisix-config","serverAddrs":["http://10.244.0.3:9180"],"tlsVerify":false}, "error": "ADC execution error for GatewayProxy/ingress-apisix/apisix-config: [ServerAddr: http://10.244.0.3:9180, Err: http://10.244.0.3:9180: PUT http://10.244.0.3:9180/apisix/admin/configs, responded with status 400 Bad Request, error_msg: secrets_conf_version must be greater than or equal to (1)]"}
2026-07-22T09:14:48.240Z ERROR provider.client client/client.go:247 failed to sync resources {"name": "GatewayProxy/ingress-apisix/apisix-config", "error": "ADC execution errors: [ADC execution error for GatewayProxy/ingress-apisix/apisix-config: [ServerAddr: http://10.244.0.3:9180, Err: http://10.244.0.3:9180: PUT http://10.244.0.3:9180/apisix/admin/configs, responded with status 400 Bad Request, error_msg: secrets_conf_version must be greater than or equal to (1)]]"}
2026-07-22T09:14:48.240Z ERROR provider apisix/provider.go:282 failed to sync {"error": "failed to sync 1 configs: GatewayProxy/ingress-apisix/apisix-config"}
2026-07-22T09:14:56.241Z INFO provider.client client/client.go:214 syncing all resources
2026-07-22T09:14:56.254Z ERROR provider.executor client/executor.go:420 ADC Server sync failed {"result": {"status":"all_failed","total_resources":0,"success_count":0,"failed_count":1,"success":[],"failed":[],"endpoint_status":[{"server":"http://10.244.0.3:9180","success":false,"reason":"PUT http://10.244.0.3:9180/apisix/admin/configs, responded with status 400 Bad Request, error_msg: secrets_conf_version must be greater than or equal to (1)"}]}, "error": "ADC Server sync failed (standalone mode): http://10.244.0.3:9180: PUT http://10.244.0.3:9180/apisix/admin/configs, responded with status 400 Bad Request, error_msg: secrets_conf_version must be greater than or equal to (1)"}
2026-07-22T09:14:56.254Z ERROR provider.executor client/executor.go:153 failed to run http sync for server {"server": "http://10.244.0.3:9180", "error": "ServerAddr: http://10.244.0.3:9180, Err: http://10.244.0.3:9180: PUT http://10.244.0.3:9180/apisix/admin/configs, responded with status 400 Bad Request, error_msg: secrets_conf_version must be greater than or equal to (1)"}
2026-07-22T09:14:56.254Z ERROR provider.client client/client.go:306 failed to execute adc command {"config": {"name":"GatewayProxy/ingress-apisix/apisix-config","serverAddrs":["http://10.244.0.3:9180"],"tlsVerify":false}, "error": "ADC execution error for GatewayProxy/ingress-apisix/apisix-config: [ServerAddr: http://10.244.0.3:9180, Err: http://10.244.0.3:9180: PUT http://10.244.0.3:9180/apisix/admin/configs, responded with status 400 Bad Request, error_msg: secrets_conf_version must be greater than or equal to (1)]"}
2026-07-22T09:14:56.255Z ERROR provider.client client/client.go:247 failed to sync resources {"name": "GatewayProxy/ingress-apisix/apisix-config", "error": "ADC execution errors: [ADC execution error for GatewayProxy/ingress-apisix/apisix-config: [ServerAddr: http://10.244.0.3:9180, Err: http://10.244.0.3:9180: PUT http://10.244.0.3:9180/apisix/admin/configs, responded with status 400 Bad Request, error_msg: secrets_conf_version must be greater than or equal to (1)]]"}
2026-07-22T09:14:56.255Z ERROR provider apisix/provider.go:282 failed to sync {"error": "failed to sync 1 configs: GatewayProxy/ingress-apisix/apisix-config"}
```

### Steps to Reproduce

A self-contained reproduction is attached as [apisix-secrets-repro.zip](https://github.com/user-attachments/files/30263545/apisix-secrets-repro.zip):

- `k8s/` - a kustomization that stands everything up + a Job that automates steps 3–6
below and prints the controller's own rejection log.
- `compose/api-driven/` - docker/podman baseline: APISIX resolves `$secret://vault/...`
in API-driven standalone (shows the gap is controller-side, not APISIX).
- `compose/file-driven/` - docker/podman baseline: the documented `apisix.yaml`
`secrets:` block resolves, but that mode has no Admin API for the controller to drive.

To reproduce the deadlock manually against the controller:

1. Deploy APISIX in API-driven standalone mode (`deployment.role: traditional`,
`role_traditional.config_provider: yaml`) + APISIX Ingress Controller 2.1.0 — e.g.
the `apisix` Helm chart 2.16.0 with `ingress-controller.enabled: true`,
`ingress-controller.config.provider.type: apisix-standalone`, and
`ingress-controller.gatewayProxy.createDefault: true`.
2. Apply any `ApisixRoute`; confirm the controller syncs it —
`GET /apisix/admin/configs` shows the route and `.secrets` is empty (`secrets = 0`).
3. Register a `secrets` manager **out of band** via the standalone Admin API (echo the
existing `*_conf_version` values back, omit `secrets_conf_version`, add the manager):
```
PUT /apisix/admin/configs
{ ...existing *_conf_version..., "secrets": [
{ "id": "vault/oidc", "uri": "http://vault.example:8200", "prefix": "kv/apisix", "token": "root" } ] }
```
Confirm `.secrets` now has 1 entry (`secrets = 1`); APISIX's `secrets_conf_version`
is now `1`.
4. Trigger a controller re-sync (e.g. `kubectl annotate apisixroute poke=$(date +%s) --overwrite`).
5. Watch the controller logs: every sync now fails with
`400 ... secrets_conf_version must be greater than or equal to (1)`.
6. Observe the blast radius: the out-of-band manager is **not** removed, and because the
push is atomic, no route reconciles — `curl` against the gateway returns `404` even
for routes unrelated to secrets.

### Environment

APISIX Ingress controller version (`apisix-ingress-controller version --long`): **2.1.0**
```Version: 2.1.0
Git SHA:
Go Version: go1.24.13
Building OS/Arch: /
Running OS/Arch: linux/amd64
```

Kubernetes cluster version (`kubectl version`):
```
Client Version: v1.33.13
Kustomize Version: v5.6.0
Server Version: v1.35.1
```
(reproduced on minikube, Podman driver)

APISIX: **3.17.0**; installed via the `apisix` Helm chart **2.16.0**;
`provider.type: apisix-standalone`

Contributor guide

No contributing guide indexed for this repository

Research direction

Trace standalone synchronization from client/client.go and client/executor.go through apisix/provider.go, focusing on construction of the full /apisix/admin/configs payload and secrets_conf_version handling. Reproduce the attached Kubernetes scenario, then verify that a pre-existing out-of-band secrets manager no longer prevents unrelated resources from syncing and that the manager remains intact.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.