microsoft / microsoft/Huabu

refactor(agenetes): decouple persistence ports from storage placement

Open
#74 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
TypeScript
Stars
146
Forks
20
Avg merge
2d 6h
Merged PRs (30d)
43

Description

Summary

Refactor Agenetes persistence so its function-specific stores remain domain-owned ports while storage placement and backend configuration move into host-injected adapters.

As part of this refactor, restore WorkloadSpec.namespace to a plain string ID. Agenetes should understand the logical namespace used for isolation and persistence addressing, but it should not see storage.root, absolute paths, or backend-specific configuration.

Scheduling and dependencies

Implementation is paused until Huabu's host storage composition and Workspace-lifecycle boundaries stabilize.

Dependency order:

  1. Merge #63 (multi-backend storage Phase 3).
  2. Merge #77 (Phase 4 aggregate structured writes, Disk recovery, Workspace leases, and retained-handle guards).
  3. Settle the Phase 5 adapter/composition follow-up far enough to define backend factory registration, init / health / close lifecycle, Workspace-to-connection binding, and how one structured backend family constructs both Huabu and Agenetes adapters. This does not require every planned backend to ship; the first real non-Disk adapter or an accepted equivalent contract is sufficient.
  4. Rebase and implement this issue as the provisional Phase 6 described by #77's storage roadmap.

The implementation must reuse the accepted host composition, Workspace lease, and retained-handle patterns rather than introduce an independent Agenetes remount policy. In particular, Workspace switching behavior and async AgentHandle.onState semantics require explicit review before broadening those shared runtime contracts.

An exploratory local spike was intentionally discarded without a commit or PR after it exposed these unresolved dependencies.

Current problem

Agenetes already owns three useful persistence ports:

  • ThreadStore for durable workload records and AgentStateSnapshot
  • EventLogStore for the Tier-1 append-only event WAL
  • TurnStore for Tier-2 folded turns and fences

However, their production implementations are synchronous and directly derive files from Namespace.storage.root. This creates several issues:

  • physical Disk placement leaks into WorkloadSpec and protocol types;
  • callers must construct { name, storage } objects instead of supplying a namespace ID;
  • synchronous ports cannot be implemented naturally by Postgres or another remote backend;
  • ThreadStore strips namespace.storage while writing and rebinds it while reading;
  • the three stores are injected separately without a shared persistence lifecycle;
  • current File adapters rely on process-local sequence and metadata caches, so their concurrency guarantees do not generalize to multi-process backends.

Huabu's StructuredStore / BlobStore split already establishes backend-family selection and lifecycle at the host composition layer. Agenetes should use compatible adapters without moving ownership of L2 persistence into CanvasStore or exposing a generic KV/SQL API.

Desired architecture

Keep narrow, function-specific Agenetes contracts and compose them as one persistence bundle:

interface AgenetesPersistence {
  namespace(id: string): NamespacePersistence;
}

interface NamespacePersistence {
  readonly threads: ThreadStore;
  readonly events: EventLogStore;
  readonly turns: TurnStore;
}

The exact aggregate shape may differ, but these constraints should hold:

  • WorkloadSpec.namespace is a string ID.
  • Store ports accept logical namespace IDs or are already namespace-scoped.
  • Store methods are asynchronous.
  • Store contracts express domain operations, not paths, JSONL, SQL, or generic transaction callbacks.
  • Disk roots are resolved by File adapter configuration supplied by the host.
  • SQLite/Postgres adapters can interpret the same namespace ID without emulating a filesystem path.
  • Agenetes packages do not import Huabu Server modules.

For example, Disk placement may be configured as:

new FileAgenetesPersistence({
  resolveNamespaceRoot: namespace => historyDirForCanvas(namespace),
});

The resolver or scoped adapter must bind the relevant Workspace identity. It must not dynamically resolve against a mutable global "current Workspace" on every operation, because a retained handle could otherwise write into a newly activated Workspace.

Work

  1. Add reusable contract suites for ThreadStore, EventLogStore, and TurnStore, covering namespace isolation, ordering, missing values, malformed records, restart behavior, and concurrency guarantees.
  2. Change WorkloadSpec.namespace and related runtime identity types from Namespace objects to string IDs.
  3. Remove namespace.storage stripping/rebinding from persisted thread records.
  4. Move namespace-to-Disk-root resolution into host-provided File adapter configuration.
  5. Make the persistence ports and Agenetes instance call chain asynchronous; do not add a blocking facade for future database adapters.
  6. Introduce a persistence aggregate or equivalent lifecycle boundary while preserving narrow domain stores.
  7. Preserve the existing threads.json and chat_v2/*.jsonl Disk layout through compatible adapters and migrations.
  8. Wire Huabu's composition root to construct the Agenetes adapters for the selected structured backend family.
  9. Update architecture documentation and migration tests.

Agent Team registry persistence is host-global rather than per-Space. It may reuse backend infrastructure and contract patterns later, but it should remain a separate persistence aggregate.

Acceptance criteria

  • WorkloadSpec.namespace is a string ID and contains no storage locator.
  • Public Agenetes persistence ports expose no absolute path or backend configuration.
  • ThreadStore, EventLogStore, and TurnStore support asynchronous implementations.
  • The Disk adapter preserves existing durable workload records and conversation history.
  • Workspace switching cannot redirect retained Agenetes handles to another Workspace.
  • Adapter contract suites pass for in-memory and Disk implementations.
  • Huabu constructs Agenetes persistence through its host composition boundary rather than rebuilding namespace storage objects at call sites.
  • The design remains compatible with the global threadId addressing direction in #58.

Non-goals

  • Replacing function-specific ports with a generic get / put storage API.
  • Moving Agenetes persistence ownership into CanvasStore.
  • Implementing every SQLite/Postgres schema in the first refactor.
  • Combining Agent Team registry state with per-namespace conversation persistence.

Related: #58, #60

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the accepted storage work in #63 and #77, then examine the ThreadStore, EventLogStore, and TurnStore contracts and the Huabu composition root. Use the contract suites, migration tests, and existing threads.json and chat_v2/*.jsonl layout as the verification points. Implementation should wait until the listed host-composition and Workspace-lifecycle boundaries are settled.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend, backend-api-design, databases
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.