privacy/classify: over-limit and over-concurrency requests both return generic 502, making them indistinguishable from an outage

We run openai/privacy-filter via cloud-api.near.ai/v1/privacy/classify as a PII backstop over agent traces. Three behaviours have been expensive to diagnose because they are all reported the same way. All figures below are measured against the live endpoint.

1. Over-limit requests return a generic 502, not 413 or 429

The limit appears to be total input tokens per request. Exceeding it returns a 502 with a short generic body -- no 413, no 429, no Retry-After. An over-limit request is therefore indistinguishable from a vendor outage by status code alone, so a client cannot tell "split this input" from "back off and retry".

Failures begin around 3,000 tokens and are total by 6,000. There also seems to be a time-varying component: 13,500 tokens succeeded one morning and 9,000 failed that afternoon.

2. Concurrency limits appear to be per API key, with no 429, and the trip briefly poisons subsequent traffic

Same payload, same key, only the timing varied:

Four requests that pass when spread over 18 seconds fail simultaneously, so this is about overlap rather than count or size. Failures are immediate rather than timeouts.

The trip then briefly affects subsequent serial requests on the same key -- including requests issued from a separate process, which is how we established it is enforced per key rather than per connection. A 429 with Retry-After would make this trivially handleable client-side.

3. openai/privacy-filter reports context_length: 512

The upstream model card documents a 128k context window. The served model reports 512, which pushes clients into aggressive chunking and multiplies request count against limits (1) and (2). If the gateway splits input internally, it would help to say so and to expose the real accepted input size.

What would help, in priority order

1. Distinct status codes: 413 for over-limit input, 429 with Retry-After for rate and concurrency limits.
2. Documented limits -- maximum input tokens per request, and maximum concurrent requests per key.
3. Clarification of the effective context/input size for openai/privacy-filter.

These may well be intended limits rather than defects; the difficulty is purely that they are not distinguishable from an outage in the response. Happy to supply more detail or re-run any of these measurements.

Start by reproducing the reported cases against cloud-api.near.ai/v1/privacy/classify, including the request-size and concurrent-request patterns. Trace the endpoint's error handling and the openai/privacy-filter context reporting in the cloud-api repository. Done means over-limit and concurrency failures are distinguishable from outages, limits are documented or exposed, and the effective input size is clarified.