feat: As a user, I want a Valkey config_provider for the data-plane role, so that I can run a decoupled data plane without etcd
- Dominant language
- Lua
- Stars
- 17.1k
- Forks
- 2.9k
- Avg merge
- 3d 16h
- Merged PRs (30d)
- 63
Description
### Description
As a user running Valkey/Redis-compatible infrastructure, I want to run APISIX data-plane nodes that sync their configuration from Valkey, so that I can operate a decoupled data plane without also running an etcd cluster.
**Context**
APISIX already supports multiple `config_provider` options for the data-plane role (`etcd`, `yaml`, `json`, `xds`). This proposal adds one more — `valkey` — following the same read-only sync pattern that `json` and `xds` established. It is purely additive: etcd remains the default and nothing about the existing providers changes.
Many teams already standardize on Valkey/Redis for their infrastructure (APISIX's own redis-backed plugins are widely used against Valkey today, e.g. #13584, #12282). Letting a data plane sync config from Valkey removes etcd as a hard dependency for those deployments.
**Why Valkey can satisfy the config-provider contract**
The historical concern with non-etcd config stores (see the [FAQ](https://github.com/apache/apisix/blob/master/docs/en/latest/FAQ.md)) is the need for resumable watch streams, prefix reads, and atomic updates. Valkey now provides primitives that map cleanly onto these:
- **Change notification** → Valkey Streams (`XREADGROUP` with consumer groups): resumable, at-least-once delivery from the last processed ID, which is exactly the "watch stream" semantic etcd provides. This is stronger than the `yaml` provider's 1s polling.
- **Prefix / directory reads** → one Hash per collection (`{apisix}:routes`, `{apisix}:services`, ...): `HGETALL` returns all items in a collection atomically.
- **Revision tracking** → an application-managed counter key (`{apisix}:revision`), `INCR`ed on each change, used to synthesize the `conf_version` / `modifiedIndex` that `core.config` consumers rely on.
- Hash-tag `{apisix}` keeps keys co-located on one shard so the design is Valkey-cluster-safe.
**Proposed design (data-plane, read-only)**
- New provider module `apisix/core/config_valkey.lua` implementing the existing `core.config` read contract (`new`, `get`, `fetch_created_obj`, `close`, `init`, `init_worker`, plus `values` / `values_hash` / `conf_version` / `clean_handlers` and `filter` / `checker` / `item_schema` handling) — the same surface `config_yaml.lua` and `config_xds.lua` satisfy today.
- Connect via the existing `api7-lua-resty-redis-connector` dependency (already used by the redis plugins).
- Add `"valkey"` to the data-plane `config_provider` enum + a `valkey` connection schema (host, port, password, database, tls) in `apisix/cli/schema.lua`, with the corresponding edits in `apisix/cli/file.lua`, `apisix/cli/ops.lua`, and the `config_ready_check` allowlist in `apisix/init.lua`.
- Config gets written to Valkey by an external process (or a Valkey-backed standalone controller); the data plane only reads and watches. **Admin API writes are explicitly out of scope for this issue** — this is data-plane sync only, matching how `xds` works.
Example configuration:
```yaml
# conf/config.yaml
deployment:
role: data_plane
role_data_plane:
config_provider: valkey
valkey:
host: "valkey.example.com"
port: 6379
password: "secret"
database: 0
tls:
enabled: true
```
**Testing**
- New `t/config-center-valkey/` directory mirroring `t/config-center-yaml/` and `t/config-center-json/`, using `--- yaml_config` blocks to set the data-plane role + `config_provider: valkey`.
- Add a `valkey/valkey` service to `ci/pod/docker-compose.common.yml` and wire `t/config-center-valkey` into the corresponding test group in `.github/workflows/build.yml`.
**Scope note**
This is deliberately scoped to the data-plane read path — the pattern with clear precedent. Extending the option to the Admin API / control-plane write path is a larger, separate discussion (it would need a write-side storage abstraction, since the Admin API calls `core.etcd` directly today) and is not part of this proposal.
I'd like to gauge maintainer interest before writing code, per CONTRIBUTING.md. Does this direction sound acceptable, and are there constraints I should account for in the design? I'm happy to implement it. Thank you!
Contributor guide
Research direction
Start by reading the existing config providers in apisix/core/config_yaml.lua and apisix/core/config_xds.lua, then inspect the provider handling in apisix/cli/schema.lua, apisix/cli/file.lua, apisix/cli/ops.lua, and apisix/init.lua. Add the data-plane Valkey provider and its configuration coverage, create t/config-center-valkey/, add the Valkey service in ci/pod/docker-compose.common.yml, and wire the test group in .github/workflows/build.yml; done means the Valkey-backed read and watch path passes its integration tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- lua, redis
- Domain
- api, backend, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100