feat: LLM API retry engine with exponential backoff and gateway awareness

Problem

Code Puppy currently has no retry logic for LLM API calls. When the API returns
429 (rate limit), 529 (overloaded), 5xx server errors, or network timeouts, the
error surfaces immediately to the user with no retry attempt. The existing
RetryManager in code_puppy/mcp_/retry_manager.py has solid retry
infrastructure but it is only wired to MCP server calls, not to the
pydantic_agent.run() call in base_agent.py.

This is especially painful when running behind a corporate LLM gateway where
bandwidth is shared across many concurrent sessions and transient 429/529
errors are common.

Proposed Solution

Add a purpose-built code_puppy/llm_retry.py module that wraps the
pydantic_agent.run() call in base_agent.py with a Claude Code-grade retry
engine. The existing _run_with_streaming_retry (which only handles
UnexpectedModelBehavior / stream-ended errors) gets merged into this new
system.

Core behavior

Backoff formula (matching Claude Code's production formula):

base   = min(500ms × 2^(attempt-1), 32_000ms)
jitter = random(0, 0.25) × base          # up to +25%
delay  = base + jitter

# Retry-After header from API takes absolute priority over computed delay

Resulting delay sequence (no Retry-After, no jitter):
500ms → 1s → 2s → 4s → 8s → 16s → 32s (capped)

Default max retries: 10 (configurable via PUPPY_MAX_LLM_RETRIES env var)

Retryable errors:

Condition	Retried?	Special handling
HTTP 429 (rate limit)	Yes	Respect Retry-After header
HTTP 529 / overloaded_error	Yes	Track consecutive count → fallback model after N
HTTP 408, 409	Yes	Normal backoff
HTTP 5xx	Yes	Normal backoff
Network timeout	Yes	Normal backoff
x-should-retry: false header	No	Always respected
x-should-retry: true header	Yes	Respected
HTTP 400 (context overflow)	Yes	Adjust max_tokens, retry immediately (no sleep)
HTTP 400 (other)	No	Fatal
HTTP 401	Yes	Clear token cache, retry
asyncio.CancelledError	No	Propagate immediately

Context window overflow: Parse inputTokens and contextLimit from the 400
error message, compute available = contextLimit - inputTokens - 1_000, cap
max_tokens at max(3_000, available) and retry immediately without sleeping.

529 model fallback: After N consecutive 529 errors (configurable, default 3),
if a fallback_model is configured, raise a FallbackTriggeredError so the
caller can switch models. If no fallback is configured, raise a hard
RetryExhaustedError.

Abort-aware sleep: The sleep between retries uses asyncio.wait_for /
asyncio.Event so that an in-flight asyncio.CancelledError interrupts the
wait immediately rather than sleeping through user cancellation.

Callback hooks (new)

Two new hooks added to callbacks.py:

"api_retry_start"  # fired before each retry sleep, receives (error, attempt, delay_ms, max_retries)
"api_retry_end"    # fired after a successful attempt following retries, receives (total_attempts,)

These allow plugins to observe retry behavior, emit UI feedback, or modify
retry strategy.

Files changed

File	Change
code_puppy/llm_retry.py	New — core retry engine
code_puppy/callbacks.py	Add api_retry_start, api_retry_end phase types
code_puppy/agents/base_agent.py	Wire llm_retry.py into pydantic_agent.run() call; retire _run_with_streaming_retry

What does NOT change

• code_puppy/mcp_/retry_manager.py — untouched, MCP retry stays separate
• Plugin API — all existing callbacks remain unchanged
• Gateway-specific behavior (skip app-layer retry when gateway manages backpressure) → separate Walmart fork plugin

Testing

• Unit tests for llm_retry.py: backoff formula, retryable classification,
  Retry-After header parsing, context overflow adjustment, consecutive 529 counter
• Integration test: mock pydantic_agent.run() to throw 429 then succeed, assert
  retry was attempted with correct delay
• Verify asyncio.CancelledError during sleep propagates immediately

References

• Claude Code retry implementation reverse-engineered from v2.1.50 cli.js
• Existing RetryManager: code_puppy/mcp_/retry_manager.py
• Main call site: code_puppy/agents/base_agent.py::run_with_mcp()