Add basic automatic retry for HTTP 422 token-limit errors with strict detection and timeouts

Summary

This issue proposes implementing basic automatic retry for HTTP 422 errors caused by completion token limit exhaustion. The goal is to make jcode more robust when using OpenAI‑compatible providers that enforce strict per‑request token caps.

---

Problem

When using an OpenAI‑compatible provider with a strict token limit (e.g., 60 000 completion tokens), jcode fails with a 422 status and does not retry, even though the error response often includes a human‑readable wait time. Example:

OpenAI-compatible chat request failed
  endpoint: https://your-provider/chat/completions
  model: DeepSeek-V4-Flash
  auth: OPENAI_COMPAT_API_KEY
  status: 422 Unprocessable Entity
  response: {"error":"Token limit exceeded: used 60323, limit 60000. Retry in 18 min."}

The operation aborts immediately. A simple wait and retry would often succeed. However, retrying blindly is risky; we need strict policies.

---

Retry policy

1. Detection strictness – retry only when the 422 response body simultaneously contains:

   • A number (used tokens)
   • Another number (token limit)
   • At least one keyword: exceeded, limit, token
     This prevents false positives (e.g., malformed request errors).

2. Wait caps – to avoid indefinite hangs:

   • Per‑wait maximum: 60 seconds (even if the API suggests longer).
   • Total timeout for all retry attempts: 2 minutes (120 seconds) from the first failure.

3. Active turn – retries must not block the main thread in a busy‑wait. Implementation should use asynchronous sleep (tokio::time::sleep) within the async request handler, and the total timeout cap ensures the operation does not exceed 2 minutes. If the provider cannot be reached within that window, the last error is returned.

---

Scope of this issue

This covers:

• Detection logic (as defined above) – English keywords only.
• Extraction of wait time from:
  • retry‑after header (preferred)
  • Error message patterns (English only)
• Exponential backoff fallback (starting at 5s, doubling, capped at 60s per wait).
• Retry limit: up to 4 retries (5 total attempts), but the 2‑minute total timeout takes precedence.
• Placement: integrate into the existing request‑handling module where 429 and 401/403 are handled.

---

Out of scope for this issue

• Non-English language support – will be a separate follow‑up issue.
• Context compaction (reducing history or max_tokens) – will be a later enhancement.
• Configurable policies via config.yaml – will be a later enhancement.
• Provider‑specific overrides – will be considered later if needed.

---

Suggested implementation details

Detection logic

When a response with HTTP status 422 is received, the implementation should:

1. Parse the response body as JSON.
2. Extract the contents of the "error" field as a string.
3. Convert the string to lowercase.
4. Check that the string contains at least two different numbers – this indicates a token usage value and a limit value.
5. Check that the string contains at least one of the following English keywords: "exceeded", "limit", or "token".
6. If all conditions are met, the error is classified as a token‑limit error and is eligible for retry.
7. If any condition fails, the error is treated as a regular 422 error and is not retried.

Wait‑time extraction

Once a retryable error is detected, the implementation should determine how long to wait before the next attempt, using the following priority order:

1. Use the retry‑after HTTP header, if present. Its value should be parsed as seconds.
2. Parse the error message for time patterns, in this order:
   • Retry in X minutes
   • Retry after X seconds
   • X minutes
   • X seconds
   • X min
   • X sec
     Where X is a number. If a match is found, convert to seconds.
3. Fallback to exponential backoff if no time is found:
   • First retry: wait 5 seconds.
   • Second retry: wait 10 seconds.
   • Third retry: wait 20 seconds.
   • Fourth retry: wait 40 seconds.
   • Each wait is capped at 60 seconds.

Retry loop behaviour

The retry loop should operate as follows:

• A maximum of 5 total attempts (the original request plus up to 4 retries) is allowed.
• Before each retry, the wait time is calculated using the logic above.
• The total cumulative wait time across all retries must not exceed 120 seconds (2 minutes).
• If the cumulative wait time would exceed 120 seconds, the loop terminates and the last error is returned.
• Each retry uses tokio::time::sleep (or equivalent) to wait asynchronously without blocking the thread.
• Each retry attempt and its calculated wait time should be logged for debugging purposes.

Integration point

• This logic should be placed in the existing provider request‑handling module, alongside the existing retry logic for HTTP 429 (rate limiting) and 401/403 (authentication refresh). A good candidate is crates/jcode‑provider‑openai‑runtime/src/.

---

Why this approach works

• Strict detection avoids retrying non‑recoverable errors.
• Caps prevent jcode from freezing for minutes.
• Async sleep keeps the system responsive.
• The logic is self‑contained and does not require new dependencies.

---

Environment

• jcode version: v0.77.1
• Provider: custom OpenAI‑compatible endpoint (internal)
• Model: DeepSeek‑V4‑Flash (or any model with token caps)

---

Additional notes

• This retry logic applies only to OpenAI‑compatible providers.
• The existing 429 and 401/403 retry logic should be used as a reference.
• All time values (waits and timeout) should be configurable via constants at the module level for now; user‑facing configuration will come in a later enhancement.

---

Supersedes

This issue replaces #994 and provides a clearer, stricter specification.

---

Next steps

After this basic retry is merged, I will open follow‑up issues for:

• Non-English language support
• Context compaction (reducing history or max_tokens before retry)
• Configurable policies (via config.yaml or environment variables)

Start in crates/jcode-provider-openai-runtime/src/ alongside the existing HTTP 429 and 401/403 retry handling, and trace the OpenAI-compatible request path and response parsing. Done means strict 422 token-limit detection, capped asynchronous retries, timeout precedence, and retry logging are implemented without changing other 422 errors; verify the behavior with the repository's relevant tests.