One unregistered event kind fails the entire conversation load (should degrade per-event)

Summary

If a single persisted event fails to deserialize (e.g. its observation.kind is an unregistered/custom type), the entire conversation fails to load and is silently dropped from the server. On agent-server startup the conversation 404s and never appears in listings — even though all its other events are intact on disk. One unknown event kind takes down the whole conversation.

Root cause

EventLog._get_single_item deserializes each event strictly:

# openhands-sdk/openhands/sdk/conversation/event_store.py
evt = Event.model_validate_json(txt)   # raises on any unknown/invalid event

State reconstruction walks the branch and reads every event via this path (rebuild_view → path_to_root → EventLog[idx]). A single ValidationError there propagates all the way up. On the agent-server, the startup scan wraps the whole per-conversation load in one try/except and, on any exception, skips the conversation entirely:

# openhands/agent_server/conversation_service.py  (__aenter__ scan loop)
for conversation_dir in self.conversations_dir.iterdir():
    try:
        ...
        await self._start_event_service(stored)   # rebuild_view() reads every event
    except Exception:
        logger.exception(f"error_loading_event_service:{conversation_dir}", ...)
        # -> conversation is NOT registered; it 404s and is absent from listings

So the blast radius of one bad/unknown event is the entire conversation, and the only signal is a log line.

Reproduction (observed on 1.33.0)

A conversation used a custom tool whose observation subclasses Observation with kind="CanvasUIObservation". The type is only known once its module is imported. When the conversation resumed without that module imported (its meta.json had an empty tool_module_qualnames), 3 of ~1,200 events failed to deserialize:

pydantic_core.ValidationError: 1 validation error for Event
observation
  Value error, Unknown kind 'CanvasUIObservation' for openhands.sdk.tool.schema.Observation;
  Expected one of: ['MCPToolObservation', 'FinishObservation', ... 'WorkflowObservation']

Result: error_loading_event_service, the conversation 404s and is missing from the panel, despite 1,197 perfectly valid events (and a healthy event tree).

Why this is worse than it looks

• Silent and total. A user sees a conversation vanish, not "3 events couldn't render."
• Fragile across versions/plugins. Any custom/plugin observation or action type, or any forward-compatible event added by a newer writer, can brick a whole conversation on a reader that doesn't know the type — even if that event is irrelevant to continuing the work.
• Recoverable data, unrecoverable UX. The bytes are fine on disk; only the strict loader makes them unreachable.

Suggested direction

Fail soft on a per-event basis instead of failing the whole conversation:

1. Tolerant deserialization for unknown kinds. When an event's kind (or nested observation/action kind) is unregistered, fall back to a generic/opaque event that preserves the raw payload and renders as text, rather than raising. (A UnknownEvent / passthrough type, similar to how some systems keep unrecognized fields.)
2. Or: isolate per-event failures during view rebuild. If a single event can't be parsed, log it, skip/placeholder it, and continue building the branch — so one bad event can't strand the conversation.
3. At minimum, don't drop the whole conversation at the server scan. Register it read-only / degraded and surface the parse error, instead of a bare 404 + log line.

Relatedly, the trigger in our case was a missing tool_module_qualnames entry on resume (the module that defines the custom type wasn't re-imported). Persisting/replaying tool-module registrations reliably would remove one common source of unknown kinds — but the loader should still degrade gracefully regardless, since custom types and cross-version events are expected in the wild.

Environment: openhands-sdk / openhands-agent-server 1.33.0.

Start with openhands-sdk/openhands/sdk/conversation/event_store.py and _get_single_item, then trace rebuild_view, path_to_root, and EventLog[idx]. Read the aenter scan loop in openhands/agent_server/conversation_service.py. Done means one invalid or unknown event no longer prevents valid events from loading or causes the conversation to be omitted from registration and listings.