OpenHands / OpenHands/software-agent-sdk
[Bug]: Triggered skills and path rules silently stop applying after condensation
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 539
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 137
Description
Is there an existing issue for the same bug?
- I have searched existing issues and this is not a duplicate.
Bug Description
Keyword-triggered skills and path-triggered rules are injected once per conversation. The content lands on an event that condensation is allowed to forget. The "already activated" marker lives in persistent conversation state and is never reset. So after a condensation, the body can be gone from everything the LLM sees, while the marker keeps blocking re-injection. For the rest of that conversation, the guidance silently stops applying.
A concrete user story: a team encodes conventions as path rules ("always wrap repository access in try-finally") precisely because deterministic injection is the feature's promise. In a long session the rule fires on turn 40, the conversation condenses around turn 80, and from then on the agent edits matching files with the rule gone. No error is raised, and conversation state still reports the rule as activated.
Scope, plainly: this is not data loss. The event log retains the content, new conversations are unaffected, and model-invocable skills can still be re-fetched through invoke_skill if the model chooses to. The bug needs a session long enough to condense, and an activation outside the condenser's protected first events (keep_first). Those are exactly the long sessions where triggered rules matter most.
How deterministic the loss is depends on the trigger type:
- Path rules: fully deterministic.
ObservationEvent.__str__()omitsextended_content, so the rule body never reaches the summarizer at all. TheSkillvalidator forcesdisable_model_invocation=Truefor path rules, so there is noinvoke_skillfallback. Re-touching a matching file re-injects nothing (probe evidence below). - Keyword skills: permitted, not guaranteed. The body reaches the summarizer only inside a 500-character preview cap (
N_CHAR_PREVIEWinMessageEvent.__str__()), and the summary prompt does not require keeping it. Short bodies may survive a given summary; bodies past the cap are truncated before the summarizer sees them. Skills that allow model invocation keep theinvoke_skillfallback.
Expected Behavior
Please confirm the intended lifecycle contract for once-per-conversation triggered content across condensation.
If triggered skill/rule instructions are meant to stay applicable after their carrying event is forgotten, the active LLM context should retain or recover them deterministically. If they are intentionally turn-local, condensation should not leave durable activation state that blocks a later qualifying turn from retrieving them again.
This report intentionally does not prescribe whether the right mechanism is preservation, reconstruction, or reactivation.
Actual Behavior
On current main (8acbbb12dbc533a086e225908158e2dfb25dc49a), two deterministic probes (both provider-independent, using the SDK's TestLLM; scripts attached below):
Probe A — keyword skill, fully public path. The skill activates on a later turn (view index 5, outside the keep_first=4 protected prefix), and condensation runs through the public Conversation.condense() API:
[before] activated_knowledge_skills = ['python_tips']
[before] sentinel present in str(view.events): True
[after] activated_knowledge_skills = ['python_tips']
[after] sentinel present in str(view.events): False
[next] last user MessageEvent.activated_skills = []
[next] last user MessageEvent.extended_content = []
Probe B — path rule, summarizer input recorded. A TestLLM subclass records the exact messages the condenser LLM receives. The observation's tool result reaches the summarizer; the rule body never does:
[setup] rule.disable_model_invocation (forced by Skill validator) = True
[before] activated_path_rules = ['style_rule']
[before] sentinel in obs.extended_content: True
[before] sentinel in str(obs) (what the summarizer is fed): False
[summarizer] call 0: sentinel present in input: False
[summarizer] call 0: 'file contents here' (obs result) present: True
[reinject] second touch obs.extended_content = []
After condensation the rule stays in activated_path_rules, the body is absent from the view, and touching a matching path again injects nothing.
Command: OPENHANDS_SUPPRESS_BANNER=1 uv run python <probe>.py
Steps to Reproduce
- Build current
OpenHands/software-agent-sdkmain withmake build. - Run probe A (keyword skill): filler messages, then a matching message so the skill activates outside the
keep_firstprefix, then publicConversation.condense()with aTestLLMsummarizer whose valid summary omits the sentinel. Observe the outputs above. - Run probe B (path rule): a file touch activates the rule through the production injection callback; the recording summarizer LLM proves the rule body never enters its input; a second touch after condensation injects nothing.
Acceptance Criteria
- The intended lifecycle of triggered skill and path-rule instructions across condensation is documented in code/tests.
- A deterministic test covers condensation forgetting the event that carried triggered skill/rule content.
- The test verifies that once-per-conversation deduplication cannot leave applicable content both absent from the active view and unavailable for deterministic recovery.
- Existing behavior for ordinary conversation summarization and explicitly invoked skills remains covered.
- The
activated_path_rules/ObservationEvent.extended_contentlifecycle is either covered by the same contract or explicitly tracked separately.
Installation Method
Editable repository checkout built with make build / uv.
SDK Version
1.42.1; upstream main commit 8acbbb12dbc533a086e225908158e2dfb25dc49a.
Version Confirmation
- Confirmed against the latest
upstream/mainavailable on 2026-08-19.
Python Version
3.13.12
Model Name
TestLLM only; deterministic and provider-independent. The scripted summary models an outcome the current summarization contract permits. The path-rule half does not depend on summarizer behavior at all, since the body never enters its input.
Operating System
Linux
Minimal Code Sample
probe_a_keyword_condense.py
"""Probe A — keyword-triggered skill loss through the PUBLIC Conversation.condense().
Differences from the reference repro:
- Condensation goes through the public `conversation.condense()` entry point
(LocalConversation.condense -> CondensationRequest event -> Agent.step ->
prepare_llm_messages -> condenser.condense -> on_event(Condensation)).
- keep_first=4, matching the production default used by
`default_condenser()` (_DEFAULT_KEEP_FIRST = 4 in
llm_summarizing_condenser.py). Note: the LLMSummarizingCondenser class field
default itself is keep_first=2; the "default condenser" factory used by the
default agent pins 4. We pass 4 explicitly.
- The skill activates on a LATER turn (outside the protected keep_first
prefix), so the failure is not an artifact of keep_first=1.
"""
from openhands.sdk.agent import Agent
from openhands.sdk.context.agent_context import AgentContext
from openhands.sdk.context.condenser import LLMSummarizingCondenser
from openhands.sdk.context.condenser.llm_summarizing_condenser import (
_DEFAULT_KEEP_FIRST,
_DEFAULT_MAX_SIZE,
)
from openhands.sdk.conversation import Conversation
from openhands.sdk.event import MessageEvent
from openhands.sdk.llm import Message, TextContent
from openhands.sdk.skills import KeywordTrigger, Skill
from openhands.sdk.testing import TestLLM
sentinel = "PRESERVE_EXACT_SENTINEL_7F3A"
skill = Skill(
name="python_tips",
content=sentinel,
source="python-tips.md",
trigger=KeywordTrigger(keywords=["python"]),
)
class_keep_first_default = LLMSummarizingCondenser.model_fields["keep_first"].default
print(f"[setup] LLMSummarizingCondenser class-field default keep_first = {class_keep_first_default}")
print(f"[setup] default_condenser() factory uses keep_first={_DEFAULT_KEEP_FIRST}, max_size={_DEFAULT_MAX_SIZE}")
print("[setup] probe passes keep_first=4 explicitly (production default via default_condenser)")
agent_llm = TestLLM.from_messages([], model="agent-test-model")
summary_llm = TestLLM.from_messages(
[
Message(
role="assistant",
content=[TextContent(text="USER_CONTEXT: continuing work; skill omitted")],
)
],
model="summary-test-model",
)
condenser = LLMSummarizingCondenser(llm=summary_llm, max_size=12, keep_first=4)
agent = Agent(
llm=agent_llm,
tools=[],
include_default_tools=[],
agent_context=AgentContext(skills=[skill], current_datetime="2026-01-01T00:00:00Z"),
condenser=condenser,
)
conversation = Conversation(agent=agent, visualizer=None)
# Filler BEFORE the skill turn so the skill-bearing MessageEvent lands outside
# the protected keep_first=4 prefix.
for i in range(4):
conversation.send_message(f"prefix filler turn {i}")
conversation.send_message("help with python") # skill activates here
for i in range(10):
conversation.send_message(f"trailing filler turn {i}")
def view_str(conv):
return "\n".join(map(str, conv.state.view.events))
print("\n[before] events in view (index: type : first line):")
for idx, ev in enumerate(conversation.state.view.events):
first_line = str(ev).splitlines()[0] if str(ev) else ""
print(f" [{idx}] {type(ev).__name__}: {first_line[:80]}")
skill_idx = next(
i
for i, ev in enumerate(conversation.state.view.events)
if isinstance(ev, MessageEvent) and sentinel in str(ev)
)
print(f"[before] skill-bearing MessageEvent index in view = {skill_idx} (keep_first=4 protects 0..3)")
print(f"[before] activated_knowledge_skills = {conversation.state.activated_knowledge_skills}")
print(f"[before] sentinel present in str(view.events): {sentinel in view_str(conversation)}")
# --- PUBLIC condensation path ---
conversation.condense()
print("\n[after] events in view (index: type : first line):")
for idx, ev in enumerate(conversation.state.view.events):
first_line = str(ev).splitlines()[0] if str(ev) else ""
print(f" [{idx}] {type(ev).__name__}: {first_line[:80]}")
print(f"[after] activated_knowledge_skills = {conversation.state.activated_knowledge_skills}")
print(f"[after] sentinel present in str(view.events): {sentinel in view_str(conversation)}")
# --- reactivation check ---
conversation.send_message("python again")
last = conversation.state.events[-1]
assert isinstance(last, MessageEvent)
print(f"\n[next] last user MessageEvent.activated_skills = {last.activated_skills}")
print(f"[next] last user MessageEvent.extended_content = {[c.text for c in last.extended_content]}")
ok = (
conversation.state.activated_knowledge_skills == ["python_tips"]
and sentinel not in view_str(conversation)
and last.activated_skills == []
and last.extended_content == []
)
print(f"\n[verdict] skill content gone from view AND reactivation suppressed: {ok}")
probe_b_path_rule.py
"""Probe B — path-triggered rule: capture the exact summarizer input.
Wiring under test (all production code paths):
- LocalConversation._on_event is built as
`_tree_stamping(_rules_injecting(base_callback))`
(local_conversation.py:457). `_rules_injecting` ->
`_maybe_inject_path_rules` (local_conversation.py:558) fires on every
ObservationEvent the agent emits, correlates it to its ActionEvent, reads
the action's `path`, matches PathTrigger skills via
`AgentContext.get_tool_use_suffix`, appends the rendered rule to
`ObservationEvent.extended_content`, and records the rule name in
`state.activated_path_rules` (dedup).
- Agent.step / the run loop emit ObservationEvents through exactly this
`conversation._on_event` callback, so calling it directly is the minimal
real entry point — same code path, without scripting tool calls through
TestLLM.
- Condensation goes through the PUBLIC `conversation.condense()` API.
- The summarizer LLM is a RecordingTestLLM (TestLLM subclass) that records
the exact `messages` it receives, so we can inspect what the condenser's
summarization prompt contained.
"""
from pydantic import PrivateAttr
from openhands.sdk.agent import Agent
from openhands.sdk.context.agent_context import AgentContext
from openhands.sdk.context.condenser import LLMSummarizingCondenser
from openhands.sdk.conversation import Conversation
from openhands.sdk.event import ActionEvent, MessageEvent, ObservationEvent
from openhands.sdk.llm import Message, MessageToolCall, TextContent
from openhands.sdk.skills import PathTrigger, Skill
from openhands.sdk.testing import TestLLM
from openhands.sdk.tool.schema import Action, Observation
sentinel = "PATH_RULE_SENTINEL_9C4E"
rule = Skill(
name="style_rule",
content=sentinel,
source="rules/style.md",
trigger=PathTrigger(paths=["src/**/*.py"]),
)
print(f"[setup] rule.disable_model_invocation (forced by Skill validator) = {rule.disable_model_invocation}")
print(f"[setup] match_path_trigger('src/pkg/app.py') = {rule.match_path_trigger('src/pkg/app.py')!r}")
class RecordingTestLLM(TestLLM):
"""TestLLM that records the messages of every completion call."""
_received: list[list[Message]] = PrivateAttr(default_factory=list)
def completion(self, messages: list[Message], **kwargs):
self._received.append(messages)
return super().completion(messages, **kwargs)
class FakeEditAction(Action):
"""Minimal file-touching action: only `path` matters to rule injection
(_touched_rule_path reads `action.path` generically)."""
path: str
class FakeObservation(Observation):
pass
agent_llm = TestLLM.from_messages([], model="agent-test-model")
summary_llm = RecordingTestLLM.from_messages(
[
Message(
role="assistant",
content=[TextContent(text="USER_CONTEXT: continuing work; rule omitted")],
)
],
model="summary-test-model",
)
condenser = LLMSummarizingCondenser(llm=summary_llm, max_size=12, keep_first=4)
agent = Agent(
llm=agent_llm,
tools=[],
include_default_tools=[],
agent_context=AgentContext(skills=[rule], current_datetime="2026-01-01T00:00:00Z"),
condenser=condenser,
)
conversation = Conversation(agent=agent, visualizer=None)
def emit_file_touch(conv, path: str, call_id: str, result_text: str) -> ObservationEvent:
"""Emit an ActionEvent/ObservationEvent pair through the conversation's
production `_on_event` callback (the one wired with `_rules_injecting`)."""
action_event = ActionEvent(
thought=[TextContent(text=f"editing {path}")],
action=FakeEditAction(path=path),
tool_name="file_editor",
tool_call_id=call_id,
tool_call=MessageToolCall(
id=call_id,
name="file_editor",
arguments=f'{{"path": "{path}"}}',
origin="completion",
),
llm_response_id=f"resp-{call_id}",
)
conv._on_event(action_event)
obs = ObservationEvent(
observation=FakeObservation.from_text(result_text),
action_id=action_event.id,
tool_name="file_editor",
tool_call_id=call_id,
)
conv._on_event(obs)
# _maybe_inject_path_rules returns a copied event; fetch the stored one.
stored = conv.state.events[-1]
assert isinstance(stored, ObservationEvent)
return stored
# Filler BEFORE the rule touch so the ObservationEvent lands outside the
# protected keep_first=4 prefix.
for i in range(4):
conversation.send_message(f"prefix filler turn {i}")
obs1 = emit_file_touch(conversation, "src/pkg/app.py", "call_1", "file contents here")
for i in range(8):
conversation.send_message(f"trailing filler turn {i}")
obs_idx = next(
i for i, ev in enumerate(conversation.state.view.events) if ev.id == obs1.id
)
print(f"\n[before] rule-carrying ObservationEvent index in view = {obs_idx} (keep_first=4 protects 0..3)")
print(f"[before] activated_path_rules = {conversation.state.activated_path_rules}")
print(f"[before] obs.extended_content = {[c.text for c in obs1.extended_content]}")
print(f"[before] sentinel in obs.extended_content: {any(sentinel in c.text for c in obs1.extended_content)}")
print(f"[before] sentinel in str(obs) (what the summarizer is fed): {sentinel in str(obs1)}")
print(f"[before] str(obs) = {str(obs1)!r}")
# --- PUBLIC condensation path ---
conversation.condense()
print(f"\n[after] activated_path_rules = {conversation.state.activated_path_rules}")
view_str = "\n".join(map(str, conversation.state.view.events))
print(f"[after] sentinel present anywhere in str(view.events): {sentinel in view_str}")
# --- KEY DELIVERABLE: exact summarizer input ---
print(f"\n[summarizer] completion calls recorded: {len(summary_llm._received)}")
for call_n, msgs in enumerate(summary_llm._received):
full_text = "\n".join(
c.text for m in msgs for c in m.content if isinstance(c, TextContent)
)
print(f"[summarizer] call {call_n}: {len(msgs)} message(s), {len(full_text)} chars")
print(f"[summarizer] call {call_n}: sentinel present in input: {sentinel in full_text}")
print(f"[summarizer] call {call_n}: 'file contents here' (obs result) present: {'file contents here' in full_text}")
print(f"[summarizer] call {call_n}: 'EXTRA_INFO' (rule wrapper) present: {'EXTRA_INFO' in full_text}")
# --- re-injection check: touch the same path again AFTER condensation ---
obs2 = emit_file_touch(conversation, "src/pkg/app.py", "call_2", "file contents v2")
print(f"\n[reinject] second touch obs.extended_content = {[c.text for c in obs2.extended_content]}")
print(f"[reinject] rule re-injected after condensation: {any(sentinel in c.text for c in obs2.extended_content)}")
print(f"[reinject] activated_path_rules = {conversation.state.activated_path_rules}")
view_str2 = "\n".join(map(str, conversation.state.view.events))
ok = (
conversation.state.activated_path_rules == ["style_rule"]
and sentinel not in view_str2
and not any(sentinel in c.text for c in obs2.extended_content)
and all(
sentinel
not in "\n".join(
c.text for m in msgs for c in m.content if isinstance(c, TextContent)
)
for msgs in summary_llm._received
)
)
print(f"\n[verdict] rule body never reached summarizer AND gone from view AND re-injection suppressed: {ok}")
Screenshots and Additional Context
Why this is not just "condensation is lossy": ordinary tool history gets summarized too, but the tool stays callable afterward. Here the capability itself is retired mid-conversation while state asserts it is active.
Prior discussion: the path-rule design comment in #3984 listed "re-injection policy after condensation" as an open question, with an inline annotation assuming existing skill dedup needs no condensation handling because the skill is "in active context, re-injecting if needed." The probes above show no such re-injection occurs. In #2744 (closed stale), maintainers called the 500-character preview truncation aggressive and asked for concrete examples of impact; this report supplies one.
One precision note for reviewers: LLMSummarizingCondenser.keep_first defaults to 2 as a class field, but the production default agent uses default_condenser() with keep_first=4. The probes use 4; the outcome is identical for any keep_first smaller than the carrying event's view index.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with local_conversation.py around _on_event and _maybe_inject_path_rules, then inspect MessageEvent.str, ObservationEvent.str, and LLMSummarizingCondenser. Run the two provider-independent probes with make build and uv to reproduce the lost content and suppressed reinjection. Done means the lifecycle contract is documented in tests and applicable triggered content cannot be both forgotten and blocked from deterministic recovery.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai-infra-agents, testing-qa
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100