lablup / lablup/mlxcel

fix(server): gate media parts on /v1/messages/count_tokens like the other non-generating routes

Open
#1,748 0 comments 0 reactions 0 assignees View on GitHub
area:inference priority:medium status:ready type:bug
Dominant language
Rust
Stars
467
Forks
54
Avg merge
4h 25m
Merged PRs (30d)
310

Description

## Problem

`anthropic_count_tokens` (`src/server/routes/anthropic.rs:694`) calls `prepare_chat_request_with_cache` at `:717` with no media-capability gate in front of it. Rendering is also what resolves media, so that call downloads every `image_url` payload the translated request carries. An Anthropic `image` block with `source.type: "url"` therefore gets fetched and counted on a checkpoint with no image tower, and the route never reaches a model worker, so nothing downstream refuses it either. The sibling handler `anthropic_messages` (`:101`) already runs `crate::server::media_capability_rejection` on `translated.chat_request` and renders it through `media_rejection_response` (added by PR #1740), which keeps 400 `invalid_request_error` for a combined video plus audio refusal and 501 `not_supported_error` per modality.

## Why this is filed separately

A pre-existing gap in #1451's scope, not a regression. PR #1740 (issue #1349) closed the same hole on the three other routes that render without generating: `router_front`, `/apply-template`, `/chat/completions/input_tokens` and `/v1/chat/completions/input_tokens`. This is the fourth, left out because fixing it changes behavior on a route that PR did not otherwise touch.

## Proposed fix

Insert the same `media_capability_rejection` plus `media_rejection_response` pair into `anthropic_count_tokens` immediately after `anthropic_request_to_chat`, ahead of the existing `MAX_TOOLS` guard, matching the order `anthropic_messages` uses. Depends on #1740 landing, since the helper arrives with it.

- `AnthropicContentBlock` (`src/server/types/anthropic_request.rs:181`) has no video or audio variant, so only the per-modality arm is reachable here today.
- Existing tests in `anthropic.rs` call `media_rejection_response` directly, not the handler, so a regression test needs an HTTP-level harness: reuse the `create_app` plus `oneshot` pattern in `src/server/max_tokens_route_tests.rs`.

## Acceptance criteria

- [ ] The gate runs before the render on `/v1/messages/count_tokens`.
- [ ] A text-only checkpoint sent an Anthropic `image` block gets the refusal `/v1/messages` gives, not a token count.
- [ ] No URL named by the request is fetched before the refusal.
- [ ] A test covers it.

## Verification

```bash
cargo test --workspace --profile test-fast --features metal,accelerate server::routes::anthropic
```

Contributor guide

Open the contributing guide

Research direction

Start in src/server/routes/anthropic.rs at anthropic_count_tokens and compare its ordering with anthropic_messages, especially around anthropic_request_to_chat, media_capability_rejection, media_rejection_response, and MAX_TOOLS. Reuse the create_app plus oneshot pattern in src/server/max_tokens_route_tests.rs, then run the listed cargo test command. Done means an image URL on a text-only checkpoint is refused before rendering or fetching, with the existing response shape.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
api, backend, testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.