[Feature]: ATOM OpenAI Server, OpenClaw

Suggestion Description

ATOM OpenAI Server — Gap Analysis vs vLLM / SGLang

Comparison of ATOM's OpenAI-compatible server against vLLM and SGLang to track feature parity.

Last updated: 2026-04-10 (post PR #489)

---

Current Architecture (after PR #489)

atom/entrypoints/openai/
├── __init__.py
├── api_server.py              # FastAPI app, endpoints, engine interface
├── protocol.py                # Pydantic request/response models
├── serving_chat.py            # Chat completion (streaming + non-streaming)
├── serving_completion.py      # Text completion (streaming + non-streaming)
├── reasoning.py               # <think> separation (ReasoningFilter + separate_reasoning)
└── tool_parser.py             # Tool call parsing (ToolCallStreamParser + parse_tool_calls)

tests/entrypoints/
├── test_openai_server.py      # Integration tests (GPU-dependent)
├── test_protocol.py           # Protocol model tests
├── test_reasoning.py          # Reasoning separation tests
├── test_serving_chat.py       # Chat response building tests
└── test_tool_parser.py        # Tool call parser tests

---

Feature Comparison

Endpoints

Endpoint	vLLM	SGLang	ATOM	Status
/v1/chat/completions	✅	✅	✅	Done
/v1/completions	✅	✅	✅	Done
/v1/models	✅	✅	✅	Done
/health	✅	✅	✅	Done
/v1/embeddings	✅	✅	❌	Gap
/v1/responses (Responses API)	✅	✅	❌	Gap
/tokenize /detokenize	✅	✅	❌	Gap
/metrics (Prometheus)	✅	✅	❌	Gap
/v1/rerank /v1/score	✅	✅	❌	Gap
/start_profile /stop_profile	❌	❌	✅	ATOM-only

Sampling Parameters

Parameter	vLLM	SGLang	ATOM	Status
temperature, top_p, top_k	✅	✅	✅	Done
max_tokens, stop, stream	✅	✅	✅	Done
ignore_eos	✅	✅	✅	Done
frequency_penalty	✅	✅	❌ accepted, not wired	Gap
presence_penalty	✅	✅	❌ accepted, not wired	Gap
repetition_penalty	✅	✅	❌	Gap
min_p	✅	✅	❌	Gap
logprobs / top_logprobs	✅	✅	❌	Gap
logit_bias	✅	✅	❌	Gap
n (multiple completions)	✅	✅	❌ accepted, not wired	Gap
seed (deterministic)	✅	✅	❌ accepted, not wired	Gap
min_tokens	✅	✅	❌	Gap
best_of	✅	partial	❌	Gap

Tool Calling

Feature	vLLM	SGLang	ATOM	Status
Tool call parsing	✅ multi-parser	✅ multi-parser	✅ Kimi-K2 format	Done (PR #489)
Streaming tool calls	✅	✅	✅	Done (PR #489)
tools passed to template	✅	✅	✅	Done (PR #489)
tool_choice enforcement	✅ schema-constrained	✅	❌ pass-through only	Gap
Multiple parser backends	✅ llama3, mistral, qwen, etc.	✅ qwen, deepseek, kimi, etc.	❌ single format	Gap
Pluggable parser API	✅ --tool-call-parser	✅ --tool-call-parser	❌	Gap

Structured Output / Guided Decoding

Feature	vLLM	SGLang	ATOM	Status
response_format: json_object	✅	✅	❌	Gap
JSON schema constrained	✅ xgrammar/outlines	✅ xgrammar/outlines	❌	Gap
Regex constrained	✅	✅	❌	Gap
Grammar (EBNF) constrained	✅	✅	❌	Gap

Reasoning / Thinking

Feature	vLLM	SGLang	ATOM	Status
<think> separation	✅ --reasoning-parser	✅ --reasoning-parser	✅ built-in	Done (PR #489)
Streaming reasoning_content	✅	✅	✅	Done (PR #489)
Multiple reasoning parsers	✅ deepseek, qwen3, granite	✅ deepseek, qwen3, kimi	❌ single format	Gap
Configurable enable/disable	✅	✅	✅ --default-chat-template-kwargs	Done

Multimodal

Feature	vLLM	SGLang	ATOM	Status
Text content (string + array)	✅	✅	✅	Done (PR #489)
Image input (image_url)	✅	✅	❌	Gap
Audio input	partial	partial	❌	Gap
Video input	partial	partial	❌	Gap

Serving Features

Feature	vLLM	SGLang	ATOM	Status
Multi-LoRA adapters	✅	✅	❌	Gap
Prefix caching (KV reuse)	✅ automatic	✅ radix tree	❌	Gap
Speculative decoding	✅	✅ EAGLE	❌	Gap
Batch prompt input	✅	✅	❌	Gap
Request abort on disconnect	✅	✅	❌	Gap
API key auth	✅ --api-key	✅	❌	Gap
CORS config	✅ --allowed-origins	✅	❌	Gap
Prometheus /metrics	✅	✅	❌	Gap
Dynamic weight update	❌	✅	❌	Gap

Debug & Observability

Feature	vLLM	SGLang	ATOM	Status
--request-log JSONL logging	❌	❌	✅	ATOM-only (PR #489)
Profiling endpoints	❌	❌	✅	ATOM-only
Latency in usage (ttft_s, tpot_s, latency_s)	❌	❌	✅	ATOM-only

---

Resolved in PR #489

• ✅ Modular architecture (split from monolithic openai_server.py)
• ✅ Tool calling — parse <|tool_calls_section_begin|> into OpenAI tool_calls format
• ✅ Streaming tool calls — delta.tool_calls chunks
• ✅ Pass tools to chat template for model-side tool declarations
• ✅ ChatMessage.content optional (tool messages with content=None)
• ✅ to_template_dict() preserving tool_calls, tool_call_id, name, reasoning_content
• ✅ Reasoning separation — ReasoningFilter (streaming) + separate_reasoning (non-streaming)
• ✅ reasoning_content field in responses
• ✅ --request-log debug logging
• ✅ Multimodal content format support
• ✅ extra="ignore" on request models
• ✅ OpenAI-format error responses
• ✅ max_tokens default increased to 8192

---

Remaining Gaps — Priority Order

P0 — Critical for production

1. Request abort on disconnect — When client disconnects SSE stream, ATOM continues generating, wasting GPU. vLLM/SGLang detect disconnect and cancel.

2. Sampling parameters — frequency_penalty, presence_penalty, logprobs, seed, n are accepted but ignored. Commonly used by clients.

P1 — Important for feature parity

3. Structured output / guided decoding — JSON schema, regex, grammar-constrained generation. Critical for reliable agent tool calling workflows.

4. Pluggable tool call parsers — Currently hardcoded to Kimi-K2 format. Need parser registry for Llama, Mistral, Qwen, DeepSeek, etc.

5. Prometheus /metrics — Standard observability for production deployments.

6. Prefix caching — KV cache reuse for shared prefixes. Major latency win for multi-turn conversations.

P2 — Nice to have

7. /v1/embeddings endpoint
8. /tokenize /detokenize endpoints
9. API key auth (--api-key)
10. CORS configuration (--allowed-origins)
11. Multi-LoRA adapter support
12. Vision/multimodal input (image_url)

---

Operating System

No response

GPU

No response

ROCm Component

No response

---

Proposal: Multi-Backend Tool Call Parser

Based on analysis of vLLM (18+ parsers) and SGLang (16+ parsers) architectures.

Current State

ATOM has a single hardcoded parser for Kimi-K2 format (<|tool_calls_section_begin|>...). Both serving_chat.py and tool_parser.py are tightly coupled to this format.

Proposed Architecture

1. Abstract Base Class

# atom/entrypoints/openai/tool_parser.py

class BaseToolCallParser(ABC):
    """Base class for model-specific tool call parsers."""

    def __init__(self, tokenizer):
        self.tokenizer = tokenizer

    @abstractmethod
    def parse_tool_calls(self, text: str) -> Tuple[str, List[ToolCall]]:
        """Non-streaming: extract tool calls from completed text.
        Returns (content_text, list_of_tool_calls).
        """
        ...

    @abstractmethod
    def create_stream_parser(self) -> "BaseStreamToolCallParser":
        """Factory method for stateful streaming parser."""
        ...


class BaseStreamToolCallParser(ABC):
    """Stateful streaming parser for tool call tokens."""

    @abstractmethod
    def process(self, text: str) -> List[Tuple[str, Any]]:
        """Process a text chunk. Returns list of (event_type, data) tuples.
        Event types: "content", "tool_call_start", "tool_call_args", "tool_call_end"
        """
        ...

    @abstractmethod
    def flush(self) -> List[Tuple[str, Any]]:
        """Flush remaining buffer content."""
        ...

This mirrors vLLM's ToolParser and SGLang's BaseFormatDetector, simplified for ATOM's needs.

2. Parser Registry

TOOL_CALL_PARSERS: Dict[str, Type[BaseToolCallParser]] = {}

def register_tool_parser(name: str):
    """Decorator to register a tool call parser."""
    def wrapper(cls):
        TOOL_CALL_PARSERS[name] = cls
        return cls
    return wrapper

3. CLI Integration

python -m atom.entrypoints.openai_server \
    --model <model> \
    --tool-call-parser kimi  # or: llama3, qwen, deepseek, mistral, hermes

If omitted, no tool call parsing is applied (current behavior for non-tool models). Unlike vLLM/SGLang, we do not need auto-detection — the user knows which model they're serving.

4. Parser Implementations (Priority Order)

Parser	Models	Token Format	Priority
kimi	Kimi-K2	<|tool_calls_section_begin|>...<|tool_call_begin|>functions.NAME:INDEX<|tool_call_argument_begin|>ARGS<|tool_call_end|>...<|tool_calls_section_end|>	P0 (exists)
qwen	Qwen 2.5/3	<tool_call>{"name": "...", "arguments": {...}}</tool_call>	P1
hermes	Hermes, NousResearch	<tool_call>{"name": "...", "arguments": {...}}</tool_call> (same XML format as Qwen)	P1 (shared impl with qwen)
llama3	Llama 3.x	<|python_tag|>{"name": "...", "parameters": {...}} or {"type": "function", "function": {...}} JSON array	P1
deepseek	DeepSeek V3/R1	<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function_name\n + JSON args + <｜tool▁call▁end｜><｜tool▁calls▁end｜> (fullwidth/block chars)	P2
mistral	Mistral/Mixtral	[TOOL_CALLS] [{"name": "...", "arguments": {...}}] JSON array	P2
granite	IBM Granite	<|tool_call|>{"name": "...", "arguments": {...}}	P2

Note: Qwen and Hermes use identical <tool_call> XML format — a single parser class can serve both, registered under two names.

5. Files to Change

File	Change
tool_parser.py	Add BaseToolCallParser, BaseStreamToolCallParser, TOOL_CALL_PARSERS registry. Refactor current Kimi code into KimiToolCallParser class.
parsers/ (new dir)	Optional: one file per parser (e.g., kimi.py, qwen.py, llama3.py). Or keep all in tool_parser.py if small.
serving_chat.py	Replace hardcoded from .tool_parser import parse_tool_calls, ToolCallStreamParser with registry lookup via TOOL_CALL_PARSERS[args.tool_call_parser].
api_server.py	Add --tool-call-parser CLI argument. Pass parser instance to chat handler.
tests/	Add per-parser unit tests with real model output samples.

6. Design Decisions (vs vLLM/SGLang)

Decision	vLLM	SGLang	ATOM Proposal
Base class	ToolParser (heavy, tied to tokenizer IDs)	BaseFormatDetector (text-based)	Text-based like SGLang — simpler, no token-ID dependency
Registry	ToolParserManager with plugin support	Dict mapping	Simple dict — no plugin system needed yet
Auto-detection	No	No	No — explicit --tool-call-parser flag
Streaming granularity	Token-by-token argument streaming	Token-by-token	Chunk-level (current approach) — simpler, sufficient for ATOM
Reasoning integration	Separate --reasoning-parser	Combined detector	Keep separate — reasoning and tool parsing are orthogonal

Implementation Effort

• Phase 1 (small): Refactor existing Kimi parser into BaseToolCallParser subclass + registry. Add --tool-call-parser flag. ~2 files changed.
• Phase 2 (medium): Add Qwen/Hermes parser (shared implementation). ~1 new parser, tests.
• Phase 3 (medium): Add Llama 3.x parser. ~1 new parser, tests.
• Phase 4 (optional): Add DeepSeek, Mistral, Granite parsers as needed.

Start by reading atom/entrypoints/openai/tool_parser.py and serving_chat.py, then review tests/entrypoints/test_tool_parser.py and test_serving_chat.py. The proposal calls for a parser abstraction, registry, CLI selection, and parser implementations, so completion requires agreeing on scope and adding tests for the supported model formats.