✨ Add LiteLLM Proxy integration

Summary

Integrate zae-limiter with LiteLLM Proxy for server-side rate limiting of LLM API calls.

Motivation

LiteLLM Proxy is a popular way to run a unified LLM gateway. Users need:

• Per-user/org token budgets
• Post-hoc adjustment (actual tokens vs estimated)
• Hierarchical limits (user → org → global)

zae-limiter's adjust() capability is a perfect fit for LLM workloads where token count is unknown upfront.

Use Case

End Users → LiteLLM Proxy (+ zae-limiter) → OpenAI/Anthropic/etc
                  │
                  └── Rate limit per user before proxying

Current LiteLLM Extension Points

LiteLLM has a CustomLogger callback system with hooks:

• async_pre_call_hook - Before LLM call (acquire lease)
• async_post_call_success_hook - After success (adjust with actual tokens)
• async_post_call_failure_hook - After failure (rollback)

Note: Using CustomLogger for rate limiting is janky - need to investigate better integration points.

Investigation Needed

• Review LiteLLM source for cleaner extension points
• Check if LiteLLM has a proper rate limiter plugin interface
• Evaluate contributing upstream vs standalone integration
• Understand how litellm_call_id or request context flows through hooks
• Test lease lifecycle management across async hooks

Rough Implementation Sketch

from litellm.integrations.custom_logger import CustomLogger
from zae_limiter import RateLimiter, RateLimitExceeded

class ZAELimiterHandler(CustomLogger):
    def __init__(self, limiter: RateLimiter, limits: list[Limit]):
        self.limiter = limiter
        self.limits = limits
        self._leases = {}  # Track by request_id - janky!
    
    async def async_pre_call_hook(self, user_api_key_dict, cache, data, call_type):
        # Acquire lease, estimate tokens
        # Reject with 429 if RateLimitExceeded
        ...
    
    async def async_post_call_success_hook(self, data, response, ...):
        # Adjust lease with actual token count
        # Release lease
        ...

Open Questions

1. Is CustomLogger the right extension point, or is there a better one?
2. How to cleanly pass lease context between pre/post hooks?
3. Should this be upstreamed to LiteLLM or standalone?
4. What config format should users use?

Acceptance Criteria

• Works with LiteLLM Proxy server-side deployment
• Pre-call rate limit check (reject before LLM call)
• Post-call adjustment with actual token usage
• Clean integration (not hacky CustomLogger abuse)
• Configuration via litellm_config.yaml or similar
• Documentation with deployment example
• Error handling (429 responses with retry_after)

Resources

• LiteLLM Call Hooks
• LiteLLM Budgets & Rate Limits
• LiteLLM Dynamic Rate Limiting

Related

• #150 - Repository Protocol (prerequisite for stable API)
• Post-1.0.0 - Need stable API before engaging with LiteLLM community

Start by reviewing LiteLLM's async_pre_call_hook, async_post_call_success_hook, and async_post_call_failure_hook, along with the linked call-hook and rate-limit resources. Investigate the cleaner extension point and request-context flow before choosing an integration approach. Done means a server-side integration supports pre-call rejection, post-call adjustment, failure rollback, configuration, documentation, and retry_after errors.