zeroae / zeroae/zae-limiter

✨ Add acquire mode for administrative block/bypass control

Open
#311 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

api-design area/limiter
Dominant language
Python
Stars
0
Forks
0
Avg merge
6h 51m
Merged PRs (30d)
104

Description

Status

BLOCKED shipped early as ADR-125 disabled in v0.12.0 (#438). This issue now covers only what remains: BYPASSED (always admit, do not subtract tokens, still count consumption) and, if still wanted, a system-level block.

What ADR-125 delivered, and how it differs from the BLOCKED sketch below:

This issue's BLOCKED sketch What shipped (ADR-125, v0.12.0)
acquire_mode attribute on config items Tri-state disabled attribute (true / false / absent = inherit) on resource and entity config items
Resolved from cached config on the hot path Uncached Repository.resolve_disabled() on the slow path only; fast path enforced by a disabled stamp on bucket items and attribute_not_exists(disabled) in the speculative UpdateItem condition. Disabling is eager: config write + fan-out stamp over every existing bucket
System > Resource > Entity hierarchy Entity(resource) > Entity(_default_) > Resource only. No system-level block (ADR-125 scopes it out)
RateLimitExceeded with reason="blocked", retry_after_seconds=None Dedicated ResourceDisabled (direct ZAELimiterError subclass, not a RateLimitError, no retry hint — 403 semantics, not 429). Always propagates regardless of on_unavailable
set_*_acquire_mode / get_* / delete_* methods disable_resource() / enable_resource() / clear_resource_disabled(), disable_entity() / enable_entity() / clear_entity_disabled() on Repository; disabled= keyword on set_resource_defaults() / set_limits()
system|resource|entity set-acquire-mode CLI zae-limiter resource|entity disable|enable|clear-disabled CLI; disabled: key in the declarative YAML manifest (#405) and Disabled property on Custom::ZaeLimiterLimits

Any BYPASSED design must be an extension of ADR-125's tri-state, not a parallel acquire_mode attribute. See Design Constraints below.

Problem or Use Case

Operators need the ability to administratively control rate limit enforcement at the entity, resource, or system level, independent of token bucket state. Current use cases include:

  • Block a misbehaving entity: An API key is abusing the system; the operator wants to immediately deny all requests without waiting for buckets to drain. Shippeddisable_entity() (ADR-125).
  • Bypass limits for privileged entities: An admin user or internal service needs unlimited access to a resource while still tracking usage.
  • Emergency kill switch: Temporarily block all access to a specific resource during an incident. Shipped at resource level — disable_resource() (ADR-125). A system-level kill switch (block every resource in a namespace at once) remains open; see below.
  • Grace period for new entities: Allow a new entity to operate without rate limiting during onboarding.

Today, on_unavailable controls behavior when DynamoDB is unreachable, and ADR-125 disabled provides administrative deny, but there is no mechanism to administratively waive rate limit enforcement while still tracking usage. The only workaround is setting extremely high limits, which still consumes DynamoDB capacity for bucket operations and doesn't provide a clean audit trail.

Proposed Solution

Introduce an AcquireMode enum that controls how acquire() enforces limits:

# ORIGINAL SKETCH — superseded for BLOCKED by ADR-125; retained for history.
class AcquireMode(str, Enum):
    ENFORCE = "enforce"   # Normal: check token buckets (default)
    BLOCKED = "blocked"   # Administrative block: always deny  -> shipped as `disabled: true`
    BYPASSED = "bypassed" # Administrative bypass: always allow, track usage  -> REMAINING

Remaining scope: extend ADR-125's tri-state disabled into a three-valued mode (working name) so that the same config attribute, the same resolution walk, and the same bucket-stamp enforcement mechanism cover both deny and waive:

Stored value Meaning
absent inherit from the next level up (unchanged from ADR-125)
disabled / true administrative block (unchanged from ADR-125)
false explicit enable / carve-out (unchanged from ADR-125)
bypass new — always admit, never subtract tokens, still count consumption

Whether this is a widening of the existing disabled attribute's value set or a sibling attribute that resolve_disabled() learns to read is an implementation choice for the ADR that supersedes/extends ADR-125 — but it must be one resolution walk, not two.

Mode Behavior
Mode Token bucket (b_*_tk) Consumption counter (b_*_tc) Capacity check DynamoDB write Status
ENFORCE (inherit / false) Subtract consumed Increment Yes — can reject Yes Current behaviour
BLOCKED disabled: true Untouched Untouched N/A — reject immediately No (fast path: conditional UpdateItem fails on attribute_not_exists(disabled), 0 WCU) Shipped — ADR-125, #438, v0.12.0
BYPASSED Untouched Increment No — always succeed Yes (counter only) Open

Key design decision (unchanged): BYPASSED does NOT subtract tokens from the bucket, only increments the consumption counter. This means:

  • Usage snapshots still reflect actual consumption for bypassed entities
  • Switching from BYPASSEDENFORCE requires no bucket reset — the bucket is at its natural refill state
  • All mode transitions are clean with no reset logic needed
Design Constraints (post-ADR-125)

The BYPASSED design must answer these before implementation:

  1. One tri-state, not two attributes. bypass is a fourth value of the ADR-125 resolution walk (entity(resource) → entity(_default_) → resource; first explicit value wins). No parallel acquire_mode attribute. resolve_disabled() (or its successor) returns the effective mode; the disabled stamp on bucket items becomes a mode stamp.
  2. Fast-path condition. ADR-125 enforces deny on the speculative fast path via attribute_not_exists(disabled) in the conditional UpdateItem, with no config read. BYPASSED must state what the fast path does when the bucket carries a bypass stamp: the tk >= :consumed guard must be skipped (always admit) while the ADD b_*_tc :consumed counter increment still executes and ADD b_*_tk -:consumed is omitted. This is a different UpdateExpression shape, so the client must know the mode before building the write — which means either (a) the stamp is cached in Repository._entity_cache alongside cascade/parent_id/shard_counts and populated from the ALL_NEW / ALL_OLD return values, or (b) the bypass write is unconditional and the stamp is consulted server-side via a ConditionExpression that admits when mode = :bypass. Option (b) still subtracts tokens unless the expression is chosen per mode, so (a) is the likely shape. The design must quantify the extra RT/RCU/WCU on the first acquire per entity (cache miss) and confirm zero extra cost on the warm path.
  3. Eager fan-out on transition. Setting/clearing bypass re-stamps every existing bucket for the scope, exactly as disable_*() does today (O(buckets) writes, two-pass GSI3 discovery, same known race window). set_resource_defaults(disabled=...) / set_limits(disabled=...) accept the new value. delete_limits() / delete_resource_defaults() re-run fan-out against the newly resolved mode.
  4. Interaction with #455's declared-scope rule. #455 makes consume the declared scope of a lease: a LeaseEntry exists only for limits named in consume, on both paths. Under bypass, the counter increment (b_*_tc) must be applied to exactly the limits in consume, and lease.adjust() / consume() / release() on a bypassed lease must still update the counter (and never tk) for keys in the declared scope, and raise on undeclared keys the same way #455 specifies. Bypass does not widen or narrow the declared scope.
  5. Cascade. Each entity resolves its mode independently (unchanged). Consistent with ADR-125, an entity-level bypass on a cascading child does not extend to its parent: the parent's bucket is written with the parent's own mode. Child=bypass, Parent=enforce → child always passes, parent checked normally; if the parent rejects, no writes for either (write-on-enter invariant). Child=bypass, Parent=disabled → ResourceDisabled with entity_id = parent.
  6. Provisioner parity. src/zae_limiter_provisioner/fanout.py mirrors the async fan-out; the YAML disabled: key (and the CFN Disabled property) must accept the new value, and the per-entity-override preservation rule from ADR-125 (an apply that re-asserts an unchanged resource-level value never clobbers an out-of-band entity carve-out) must hold for bypass too.
  7. System-level block (optional, separate decision). ADR-125 scoped out system deliberately. If a namespace-wide kill switch is still wanted, it needs its own justification: set_system_defaults() has no fan-out today, and a system-level stamp touches every bucket in the namespace (GSI4 discovery). Decide whether this is in scope for #311 or a separate issue before implementation.
Configuration Hierarchy

Acquire mode is stored as an acquire_mode attribute on existing config records (no new items). Resolution follows the same four-tier hierarchy as limits: Entity (resource-specific) > Entity (default) > Resource > System > ENFORCE (default)

Superseded by ADR-125: resolution is Entity (resource-specific) > Entity (_default_) > Resource, first explicit value wins, no system level. bypass joins that walk as a new explicit value.

Hot Path

Original cached-config sketch (Phase 1 READ / Phase 2 CHECK / Phase 3 WRITE). Superseded: ADR-125 enforces on the speculative fast path via the bucket stamp with no config read. BYPASSED must follow the same pattern — see Design Constraint 2.

Cascade Behavior

See Design Constraint 5. Child=BLOCKED, Parent=ENFORCE → child rejected immediately, parent never reached. ShippedResourceDisabled raised from the child stamp; parent write never issued. Child=ENFORCE, Parent=BLOCKED → child passes, parent rejected → raise, no writes for either. ShippedResourceDisabled with entity_id = parent; child write compensated per write-on-enter.

API Surface

Python API (on Repository, matching ADR-125 placement — not RateLimiter):

Level Set Clear Status
System set_system_acquire_mode(mode) / get_system_acquire_mode() / delete_system_acquire_mode() Struck — no system level (ADR-125). Re-add only if Design Constraint 7 is accepted
Resource (block) set_resource_acquire_mode(resource, mode) / get_resource_acquire_mode / delete_resource_acquire_mode Shipped as disable_resource() / enable_resource() / clear_resource_disabled()
Entity (block) set_acquire_mode(entity_id, mode, resource=None) / get_acquire_mode / delete_acquire_mode Shipped as disable_entity() / enable_entity() / clear_entity_disabled()
Resource (bypass) bypass_resource(resource) (working name) clear_resource_disabled(resource) (existing — clears any explicit value) Open
Entity (bypass) bypass_entity(entity_id, resource=None) (working name) clear_entity_disabled(entity_id, resource=None) (existing) Open
Any set_resource_defaults(..., disabled="bypass") / set_limits(..., disabled="bypass") — or whatever the extended keyword is named Open
List list_resources_with_acquire_mode() / list_entities_by_acquire_mode(mode, resource=None) Deferred — ADR-125 shipped no list query and no sparse GSI; get_resource_defaults() / get_limits() expose the explicit value. Reconsider only if an operator use case surfaces

SyncRepository exposes matching generated methods.

CLI:

# ~~System level~~ — struck (no system level in ADR-125; see Design Constraint 7)

# Resource level
# zae-limiter resource disable|enable|clear-disabled RESOURCE_NAME   <- shipped (ADR-125)
zae-limiter resource bypass RESOURCE_NAME                              # open (working name)

# Entity level
# zae-limiter entity disable|enable|clear-disabled ENTITY_ID [--resource R]   <- shipped (ADR-125)
zae-limiter entity bypass ENTITY_ID [--resource R]                            # open (working name)

# `resource get-defaults` / `entity get-limits` already show `Status: DISABLED` /
# `Status: enabled (explicit override)`; add `Status: BYPASSED`.

Declarative manifest (#405): the existing disabled: key on resources.<name> and entities.<id>.resources.<name> accepts the new value; the CFN Disabled property round-trips it.

DynamoDB Storage

Mode stored on existing config records — no new itemsalready true for ADR-125's disabled:

Level PK SK Attribute Status
System {ns}/SYSTEM# #CONFIG acquire_mode Struck — no system level
Resource {ns}/RESOURCE#{resource} #CONFIG disabled (tri-state) Shipped — ADR-125
Entity-wide {ns}/ENTITY#{id} #CONFIG#_default_ disabled (tri-state) Shipped — ADR-125
Entity+resource {ns}/ENTITY#{id} #CONFIG#{resource} disabled (tri-state) Shipped — ADR-125
Bucket stamp {ns}/BUCKET#{id}#{resource}#{shard} #STATE disabled (present = blocked) Shipped — ADR-125
Bucket stamp same same bypass value on the same stamp (or a sibling — the ADR decides) Open

Sparse GSI4 for list queries (only indexes records where acquire_mode is set) — struck. GSI4 is now taken by namespace-scoped item discovery (purge_namespace()), ADR-125 shipped no list index, and no operator use case has asked for one.

Exception Enrichment

RateLimitExceeded gains a reason field to distinguish administrative blocks from capacity exhaustion. Superseded — ADR-125 raises a dedicated ResourceDisabled (not a RateLimitError, no retry_after_seconds) so callers map it to 403 rather than 429. Bypass never raises, so no exception change remains in scope.

Entity Model

Entity gains an acquire_mode field. Deferred — ADR-125 did not add a field to Entity; the explicit value is read from config (get_limits() / get_resource_defaults()). Reconsider only if the bypass ADR needs it.

Acceptance Criteria

Shipped as ADR-125 (#438, v0.12.0) — retained for traceability
  • Tri-state disabled stored on resource and entity config items (entity default + entity+resource)
  • Resolution walk: Entity+resource > Entity _default_ > Resource, first explicit value wins
  • Block enforced on the speculative fast path via bucket stamp + attribute_not_exists(disabled); raises ResourceDisabled before any token write
  • Cascade: child stamp checked first; parent stamp raises ResourceDisabled with entity_id = parent
  • disable_* / enable_* / clear_*_disabled on Repository and SyncRepository; disabled= keyword on set_resource_defaults() / set_limits()
  • resource|entity disable|enable|clear-disabled CLI; Status: line in get-defaults / get-limits
  • disabled: in YAML manifest and Disabled on Custom::ZaeLimiterLimits; provisioner fan-out preserves entity carve-outs
  • Audit events for disable/enable/clear at resource and entity level
  • CLAUDE.md, ADR-125, and user docs updated
Remaining — BYPASSED
Design (ADR that extends ADR-125)
  • ADR filed (new number; does not edit ADR-125) that adds bypass as a value of the existing tri-state and answers Design Constraints 1–6 above, with RT/RCU/WCU quantified for cold-cache and warm-cache acquire
  • Decision recorded on Design Constraint 7 (system-level block): in scope here, split to a new issue, or rejected
Core
  • bypass value stored on the same resource / entity config attribute ADR-125 uses (no parallel acquire_mode)
  • resolve_disabled() (or its renamed successor) returns the effective mode from the single walk; bypass is an explicit value that stops the walk
  • Eager fan-out stamps every existing bucket for the scope on set/clear, same two-pass discovery as disable_*
  • Speculative fast path under bypass: admits without the tk >= :consumed guard, omits ADD b_*_tk -:consumed, still issues ADD b_*_tc :consumed — 0 RCU, 1 WCU per bucket, no config read
  • Slow path under bypass: same write shape; _commit_initial() never subtracts tk
  • lease.adjust() / consume() / release() under bypass update b_*_tc only, for keys in the declared scope (#455); undeclared keys behave exactly as #455 specifies
  • _rollback() under bypass compensates the counter only
  • Cascade: each entity resolves independently; child bypass does not extend to the parent; parent disabled still raises ResourceDisabled(entity_id=parent)
  • Bucket TTL rules (#271/#296) unchanged by mode
  • ENFORCE (inherit / false) behaviour unchanged; disabled: true behaviour unchanged
API
  • bypass_resource(resource) / bypass_entity(entity_id, resource=None) (or the names the ADR settles on) on Repository, returning the bucket-write count like disable_*
  • Existing clear_resource_disabled() / clear_entity_disabled() clear bypass too
  • set_resource_defaults() / set_limits() accept the new value via the existing keyword; explicit pass fans out immediately
  • delete_limits() / delete_resource_defaults() re-run fan-out against the newly resolved mode (extends the ADR-125 rule)
  • SyncRepository generated methods match
CLI
  • resource bypass RESOURCE_NAME and entity bypass ENTITY_ID [--resource R]
  • resource get-defaults / entity get-limits print Status: BYPASSED
Declarative limits (#405)
  • YAML disabled: key (on resources.<name> and entities.<id>.resources.<name>) accepts the new value; Custom::ZaeLimiterLimits Disabled round-trips it
  • src/zae_limiter_provisioner/fanout.py mirrors the new stamp and preserves entity carve-outs on re-apply
Observability
  • Audit events recorded for bypass set/clear at resource and entity level ($RESOURCE:{name} / entity id, per ADR-106)
  • Usage snapshots for bypassed entities reflect real consumption (counter-driven, per issue #179 rule — no tk delta derivation)
Tests
  • Unit: fast path and slow path under bypasstk unchanged, tc incremented, 0 RCU / 1 WCU on warm path
  • Unit: resolution walk with bypass at each level, including entity bypass carve-out under a disabled resource and the reverse
  • Unit: cascade matrix (child bypass / parent enforce, child bypass / parent disabled, child enforce / parent bypass)
  • Unit: adjust() / consume() / release() / rollback under bypass touch tc only and respect #455 declared scope
  • Unit: fan-out on set/clear and on delete_* re-resolution
  • Integration (LocalStack): stamp visible on every shard; aggregator refill (#317) does not fight the stamp
  • Generated sync tests match
Documentation
  • CLAUDE.md "Disabling Resources and Entities (ADR-125)" section extended to cover bypass; DynamoDB writer table gains the bypass rows
  • docs/guide/ and docs/cli.md updated; docs-updater run
Remaining — System-level block (only if Design Constraint 7 is accepted)
  • disable_system() / enable_system() / clear_system_disabled() with namespace-wide fan-out via GSI4
  • system disable|enable|clear-disabled CLI
  • Manifest system.disabled: accepted (ADR-125 explicitly rejects this today — needs the ADR to reverse it)

Alternatives Considered

Mode as parameter on set_limits(): Simpler API surface (fewer methods) but conflates limit configuration with enforcement control. ADR-125 chose both: dedicated disable_* methods and a disabled= keyword on the setters (with a "preserve" sentinel default). Bypass follows the same pattern.

Separate block() / bypass() / unblock() API: Breaks the set/get/delete naming convention. ADR-125 went this way (disable_resource() / enable_resource() / clear_resource_disabled()); the bypass methods should match it, not the original set_*_acquire_mode sketch.

Mode on META/bucket records (read every acquire): Rejected in favour of cached config. Reversed by ADR-125: the bucket stamp is the enforcement mechanism, precisely because the speculative fast path never reads config. Cached-config resolution would have left a warm bucket admitting traffic against a disabled resource. Bypass inherits this decision.

Subtract tokens during bypass + reset on mode change: Would require bulk bucket resets when switching resource-level bypass back to enforce. The "don't subtract, only track counter" approach eliminates all reset logic and makes mode transitions clean at every level. Still the decision.

Boolean blocked flag: Too limited — doesn't support bypass. ADR-125 shipped a tri-state boolean, which supports inherit / explicit carve-outs. It is limited only in that it cannot express bypass — hence this issue's remaining scope is to widen its value set rather than add a parallel attribute.

Parallel acquire_mode attribute alongside disabled: Rejected. Two attributes with two resolution walks and two stamps would let them disagree (e.g. disabled: true at resource, acquire_mode: bypassed at entity) with no defined precedence, and would double the fan-out cost on every transition.

Related

  • ADR-125 — Resource and Entity Disable (docs/adr/125-resource-disable.md), Accepted 2026-08-30
  • #438 — PR that shipped resource/entity disable (v0.12.0)
  • #455 — consume as the declared scope of a lease; adjust() on undeclared keys
  • #405 — Declarative limits (YAML manifest + provisioner) that disabled: / bypass round-trip through
  • #309 — Write-on-enter invariant (no writes on rejection; bypass writes counter only)
  • #179 — Counter-driven consumption (bypass must not rely on tk deltas)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with docs/adr/125-resource-disable.md and the existing Repository disabled path described in the issue, then compare the async and provisioner fan-out in src/zae_limiter_provisioner/fanout.py. Define the ADR extension for bypass before implementation, including cache and bucket-stamp behavior, cascade and lease scope, and configuration/API names. Done means bypass admits requests, increments only declared consumption counters, preserves existing disabled behavior, and has matching provisioner and manifest support.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
backend, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.