apache / apache/apisix-ingress-controller
bug: standalone/ADC - an out-of-band `secrets` resource deadlocks the sync loop (`secrets_conf_version` 400), blocking all reconciliation
- Dominant language
- Go
- Stars
- 1.1k
- Forks
- 390
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 12
Description
### Current Behavior
We run APISIX in **API-driven standalone** mode driven by the ingress controller
(ADC, `provider.type: apisix-standalone`). We wanted a route's plugin to reference a
Vault secret via APISIX's native `$secret://vault/...`. There is no CRD or
`GatewayProxy` field we could find to declare the backing `secrets` secret-manager, so
the only way to get one onto the data plane is to register it directly on APISIX's
standalone Admin API (out of band).
Doing that **deadlocks the controller**. Once the out-of-band manager bumps APISIX's
`secrets_conf_version` from `0` to `1`, every subsequent controller/ADC full-config
push is rejected:
```
PUT /apisix/admin/configs → 400 Bad Request:
secrets_conf_version must be greater than or equal to (1)
```
ADC has no `secrets` resource type, so its push carries `secrets_conf_version: 0`
while APISIX is now at `1`; APISIX's monotonic version check rejects the **whole**
push. Because each standalone update replaces the full configuration **atomically**,
nothing reconciles after that: the out-of-band manager is **not** wiped, the controller
is simply **stuck**, and even unrelated routes stop applying (they 404 at the gateway).
This looks like a robustness bug independent of the secret use case: once
`secrets_conf_version` is `> 0` by **any** means (an out-of-band manager, a leftover
from a prior setup), ADC - which always sends `0` - can never sync again, and it takes
all other resources down with it.
For reference, APISIX itself handles `$secret://` fine in this mode - we confirmed it
end-to-end with two standalone docker/podman baselines (in the attached bundle), so the
problem is on the controller/ADC side, not APISIX:
- `compose/api-driven/` - in API-driven standalone, APISIX resolves `$secret://vault/...`
when a `secrets` manager is present in the `/configs` payload.
- `compose/file-driven/` - the documented `apisix.yaml` `secrets:` block resolves, but
that mode disables the Admin API the controller drives (so it can't be used together
with the controller).
### Expected Behavior
- The controller/ADC should not deadlock on a resource type it doesn't manage: it
should send a valid (or otherwise tolerated) `secrets_conf_version` so a stray or
out-of-band `secrets` resource can't block reconciliation of everything else.
- Ideally there would be a supported way to declare an APISIX `secrets` secret-manager
through the controller (e.g. a `secrets` field on `GatewayProxy`, or a dedicated
CRD), so `$secret://vault/...` references can be used. Today we can't find one.
(`secretRef` reads a Kubernetes Secret and injects plaintext - a different mechanism,
not APISIX's native `$secret://` manager.)
### Error Logs
Controller `manager` container - this block repeats on **every** sync attempt after the
out-of-band `secrets` resource exists:
```
2026-07-22T09:14:48.222Z INFO provider.client client/client.go:214 syncing all resources
2026-07-22T09:14:48.239Z ERROR provider.executor client/executor.go:420 ADC Server sync failed {"result": {"status":"all_failed","total_resources":0,"success_count":0,"failed_count":1,"success":[],"failed":[],"endpoint_status":[{"server":"http://10.244.0.3:9180","success":false,"reason":"PUT http://10.244.0.3:9180/apisix/admin/configs, responded with status 400 Bad Request, error_msg: secrets_conf_version must be greater than or equal to (1)"}]}, "error": "ADC Server sync failed (standalone mode): http://10.244.0.3:9180: PUT http://10.244.0.3:9180/apisix/admin/configs, responded with status 400 Bad Request, error_msg: secrets_conf_version must be greater than or equal to (1)"}
2026-07-22T09:14:48.240Z ERROR provider.executor client/executor.go:153 failed to run http sync for server {"server": "http://10.244.0.3:9180", "error": "ServerAddr: http://10.244.0.3:9180, Err: http://10.244.0.3:9180: PUT http://10.244.0.3:9180/apisix/admin/configs, responded with status 400 Bad Request, error_msg: secrets_conf_version must be greater than or equal to (1)"}
2026-07-22T09:14:48.240Z ERROR provider.client client/client.go:306 failed to execute adc command {"config": {"name":"GatewayProxy/ingress-apisix/apisix-config","serverAddrs":["http://10.244.0.3:9180"],"tlsVerify":false}, "error": "ADC execution error for GatewayProxy/ingress-apisix/apisix-config: [ServerAddr: http://10.244.0.3:9180, Err: http://10.244.0.3:9180: PUT http://10.244.0.3:9180/apisix/admin/configs, responded with status 400 Bad Request, error_msg: secrets_conf_version must be greater than or equal to (1)]"}
2026-07-22T09:14:48.240Z ERROR provider.client client/client.go:247 failed to sync resources {"name": "GatewayProxy/ingress-apisix/apisix-config", "error": "ADC execution errors: [ADC execution error for GatewayProxy/ingress-apisix/apisix-config: [ServerAddr: http://10.244.0.3:9180, Err: http://10.244.0.3:9180: PUT http://10.244.0.3:9180/apisix/admin/configs, responded with status 400 Bad Request, error_msg: secrets_conf_version must be greater than or equal to (1)]]"}
2026-07-22T09:14:48.240Z ERROR provider apisix/provider.go:282 failed to sync {"error": "failed to sync 1 configs: GatewayProxy/ingress-apisix/apisix-config"}
2026-07-22T09:14:56.241Z INFO provider.client client/client.go:214 syncing all resources
2026-07-22T09:14:56.254Z ERROR provider.executor client/executor.go:420 ADC Server sync failed {"result": {"status":"all_failed","total_resources":0,"success_count":0,"failed_count":1,"success":[],"failed":[],"endpoint_status":[{"server":"http://10.244.0.3:9180","success":false,"reason":"PUT http://10.244.0.3:9180/apisix/admin/configs, responded with status 400 Bad Request, error_msg: secrets_conf_version must be greater than or equal to (1)"}]}, "error": "ADC Server sync failed (standalone mode): http://10.244.0.3:9180: PUT http://10.244.0.3:9180/apisix/admin/configs, responded with status 400 Bad Request, error_msg: secrets_conf_version must be greater than or equal to (1)"}
2026-07-22T09:14:56.254Z ERROR provider.executor client/executor.go:153 failed to run http sync for server {"server": "http://10.244.0.3:9180", "error": "ServerAddr: http://10.244.0.3:9180, Err: http://10.244.0.3:9180: PUT http://10.244.0.3:9180/apisix/admin/configs, responded with status 400 Bad Request, error_msg: secrets_conf_version must be greater than or equal to (1)"}
2026-07-22T09:14:56.254Z ERROR provider.client client/client.go:306 failed to execute adc command {"config": {"name":"GatewayProxy/ingress-apisix/apisix-config","serverAddrs":["http://10.244.0.3:9180"],"tlsVerify":false}, "error": "ADC execution error for GatewayProxy/ingress-apisix/apisix-config: [ServerAddr: http://10.244.0.3:9180, Err: http://10.244.0.3:9180: PUT http://10.244.0.3:9180/apisix/admin/configs, responded with status 400 Bad Request, error_msg: secrets_conf_version must be greater than or equal to (1)]"}
2026-07-22T09:14:56.255Z ERROR provider.client client/client.go:247 failed to sync resources {"name": "GatewayProxy/ingress-apisix/apisix-config", "error": "ADC execution errors: [ADC execution error for GatewayProxy/ingress-apisix/apisix-config: [ServerAddr: http://10.244.0.3:9180, Err: http://10.244.0.3:9180: PUT http://10.244.0.3:9180/apisix/admin/configs, responded with status 400 Bad Request, error_msg: secrets_conf_version must be greater than or equal to (1)]]"}
2026-07-22T09:14:56.255Z ERROR provider apisix/provider.go:282 failed to sync {"error": "failed to sync 1 configs: GatewayProxy/ingress-apisix/apisix-config"}
```
### Steps to Reproduce
A self-contained reproduction is attached as [apisix-secrets-repro.zip](https://github.com/user-attachments/files/30263545/apisix-secrets-repro.zip):
- `k8s/` - a kustomization that stands everything up + a Job that automates steps 3–6
below and prints the controller's own rejection log.
- `compose/api-driven/` - docker/podman baseline: APISIX resolves `$secret://vault/...`
in API-driven standalone (shows the gap is controller-side, not APISIX).
- `compose/file-driven/` - docker/podman baseline: the documented `apisix.yaml`
`secrets:` block resolves, but that mode has no Admin API for the controller to drive.
To reproduce the deadlock manually against the controller:
1. Deploy APISIX in API-driven standalone mode (`deployment.role: traditional`,
`role_traditional.config_provider: yaml`) + APISIX Ingress Controller 2.1.0 — e.g.
the `apisix` Helm chart 2.16.0 with `ingress-controller.enabled: true`,
`ingress-controller.config.provider.type: apisix-standalone`, and
`ingress-controller.gatewayProxy.createDefault: true`.
2. Apply any `ApisixRoute`; confirm the controller syncs it —
`GET /apisix/admin/configs` shows the route and `.secrets` is empty (`secrets = 0`).
3. Register a `secrets` manager **out of band** via the standalone Admin API (echo the
existing `*_conf_version` values back, omit `secrets_conf_version`, add the manager):
```
PUT /apisix/admin/configs
{ ...existing *_conf_version..., "secrets": [
{ "id": "vault/oidc", "uri": "http://vault.example:8200", "prefix": "kv/apisix", "token": "root" } ] }
```
Confirm `.secrets` now has 1 entry (`secrets = 1`); APISIX's `secrets_conf_version`
is now `1`.
4. Trigger a controller re-sync (e.g. `kubectl annotate apisixroute poke=$(date +%s) --overwrite`).
5. Watch the controller logs: every sync now fails with
`400 ... secrets_conf_version must be greater than or equal to (1)`.
6. Observe the blast radius: the out-of-band manager is **not** removed, and because the
push is atomic, no route reconciles — `curl` against the gateway returns `404` even
for routes unrelated to secrets.
### Environment
APISIX Ingress controller version (`apisix-ingress-controller version --long`): **2.1.0**
```Version: 2.1.0
Git SHA:
Go Version: go1.24.13
Building OS/Arch: /
Running OS/Arch: linux/amd64
```
Kubernetes cluster version (`kubectl version`):
```
Client Version: v1.33.13
Kustomize Version: v5.6.0
Server Version: v1.35.1
```
(reproduced on minikube, Podman driver)
APISIX: **3.17.0**; installed via the `apisix` Helm chart **2.16.0**;
`provider.type: apisix-standalone`
Contributor guide
No contributing guide indexed for this repository
Research direction
Trace standalone synchronization from client/client.go and client/executor.go through apisix/provider.go, focusing on construction of the full /apisix/admin/configs payload and secrets_conf_version handling. Reproduce the attached Kubernetes scenario, then verify that a pre-existing out-of-band secrets manager no longer prevents unrelated resources from syncing and that the manager remains intact.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100