History compaction: chat.compact()

Motivation

Long-running chatlas sessions (agents, chatbots, notebooks kept open all day) eventually blow the context window, and chatlas currently has no built-in remedy beyond manually calling chat.set_turns() to prune history yourself. LangChain 1.0 ships SummarizationMiddleware, which auto-summarizes older messages once a token threshold is hit, and this pattern is now standard in agent products (Claude Code included). This is arguably the concrete design proposal for chatlas#8 ("Make it easier to impose limits on conversation size") — this issue frames a specific mechanism to close (part of) that issue.

Proposed approach

A plain method, not hidden middleware — explicit and inspectable, in keeping with chatlas's existing style (chat.set_turns(), chat.get_turns() in chatlas/_chat.py):

from chatlas import ChatAnthropic

chat = ChatAnthropic()
# ... long conversation ...

# Manual, opt-in trigger (default):
if chat.token_count(include="complete") > 100_000:
    chat.compact()  # and chat.compact_async()

# Optionally, summarize with a cheaper/faster model than the main conversation:
chat.compact(model=ChatAnthropic(model="claude-haiku-4-5"))

# Opt-in automatic triggering, off by default:
chat.compact(auto=True, threshold=100_000)  # checked at the start of each .chat() call

Implementation sketch:

• Leverage Chat.token_count()/token_count_async() (chatlas/_chat.py:671), specifically the recently-added include="complete" mode, which already estimates system prompt + .get_turns() history — exactly the heuristic needed to decide whether to compact.
• compact() selects a prefix of self._turns to summarize (e.g. all but the last N turns, or all but the last complete user/assistant pair), submits them to the provider (or a caller-supplied Chat for a cheaper model) with a summarization prompt, and replaces that prefix with a single synthetic turn pair (a UserTurn/SystemTurn note plus an AssistantTurn containing the summary as ContentText) via Chat.set_turns() (chatlas/_chat.py:357), which already supports exactly this "trim and replace" use case per its own docstring ("Clearing (or trimming) the chat history").
• Non-text content during summarization: ContentToolRequest/ContentToolResult pairs, ContentImageInline/ContentPDF, etc. (chatlas/_content.py) inside the summarized range should be described in the summary text (e.g. "user uploaded a PDF; assistant called search_docs(query=...) and got N results") rather than silently dropped, since dropping tool results that established facts the model relied on can cause it to contradict itself later. Turn.__str__ (chatlas/_turn.py:115) already gives a reasonable string form of each content type to seed this.
• AssistantTurn.cost/.tokens (chatlas/_turn.py:340,343) for the summarized turns are lost from get_tokens()/get_cost() accounting once replaced — this should be documented as expected (cost already spent isn't un-spent) and possibly preserved as metadata on the synthetic turn rather than silently discarded.

Alternatives / prior art

• Closes/addresses part of chatlas#8 directly.
• No direct ellmer equivalent found (searched tidyverse/ellmer for "compact", "summar" — no hits), so this would be new ground for the ellmer/chatlas pairing; worth raising upstream once chatlas's design is validated, per the project's alignment goal.
• Alternative considered and rejected for v1: automatic middleware-style compaction with no explicit call, à la LangChain's SummarizationMiddleware. Given YAGNI and that chatlas favors explicit methods over hidden hooks (see how set_model_params(), set_turns() are all explicit calls), default to manual/explicit triggering, with auto=True as an opt-in convenience layered on top of the same method.

Open questions

• Where should the threshold check live if auto=True — inside _chat_impl/_chat_impl_async (chatlas/_chat.py:2647,2734) before submitting turns, or as a wrapper the caller opts into explicitly?
• Should compact() support summarizing tool results into a persisted ContentToolResult-shaped stub token (to keep e.g. expand_tool_result in chatlas/_content_expand.py and dangling-request logic in chatlas/_turn.py happy) rather than plain text?
• Is a single summary turn enough, or should compaction be tiered (e.g. keep a rolling summary + last K turns, re-summarizing as needed)?
• Should the system prompt ever be folded into compaction, or always preserved verbatim (current set_turns() behavior preserves it automatically)?

Drafted from a competitive review of llm / Pydantic AI / LangChain / LiteLLM (July 2026); filed via Claude Code on behalf of @cpsievert.

Start by reading Chat.token_count() and set_turns() in chatlas/_chat.py, then inspect _chat_impl/_chat_impl_async and the content and turn types in chatlas/_content.py and chatlas/_turn.py. Resolve the open design questions around summarization boundaries, non-text content, accounting, and automatic triggering before implementation; done requires a defined compact() API and verified behavior for synchronous and asynchronous chats.