Memory.forget() races pending background saves from remember_many(): forgotten content is resurrected after drain_writes()

Description

Memory.forget() can return while a pending background save (submitted by remember_many()) still holds the very content being forgotten. When the background save lands afterwards, the "forgotten" content is resurrected and recall() returns it. forget() neither drains the pending-save queue nor takes _reset_lock, although the sibling destructive methods already do exactly that:

• recall() documents a read barrier and calls drain_writes() (unified_memory.py, recall — "Read barrier: wait for any pending background saves to finish").
• reset() / reset_all() hold _reset_lock and call drain_writes() before deleting.
• forget() (unified_memory.py:818) does neither; update() doesn't either.

The write path is genuinely asynchronous: remember_many() submits _background_encode_batch to a single-worker executor (_submit_save), and the docstring only promises a barrier for subsequent recall() calls. A caller that calls forget() immediately after remember_many() — the natural "correct the mistake" pattern — gets a return value of 0 ("nothing deleted") while the doomed write commits later, with no error and no way to observe the race.

Environment

• crewAI 1.15.20, pinned revision 143e902178a07d0f13f9db2308f983aaacaaf0f8 (editable install of lib/crewai)
• Memory + LanceDBStorage in a temporary directory; deterministic embedder fixture; no LLM is constructed (shallow recall, explicit scope/categories/metadata). No network.

Steps to Reproduce

Deterministic reproduction: the embedder blocks on a threading.Event while the background save is between submit and storage-write, so the interleaving is forced rather than lucky. Core schedule (full runnable script inline below):

from crewai.memory.storage.lancedb_storage import LanceDBStorage
from crewai.memory.unified_memory import Memory

memory = Memory(storage=LanceDBStorage(path=..., vector_dim=16), embedder=gated_embedder, llm="unused")

# 1. Background save submitted; the gated embedder holds it before the storage write.
memory.remember_many(
    ["Project Falcon ships on Friday. VG_FORGOTTEN_CANARY"],
    scope="/crew", categories=["projects"], metadata={"marker": "canary"},
    importance=0.5, source="s1",
)

# 2. Forget the same content while the save is still pending.
print(memory.forget(scope="/crew", metadata_filter={"marker": "canary"}))  # -> 0

# 3. Release the embedder; the background save now commits the "forgotten" content.
gated_embedder.gate.set()
memory.drain_writes()

# 4. Recall returns the forgotten content.
matches = memory.recall("Falcon", scope="/crew", depth="shallow", source="s1")
print([m.record.content for m in matches])
# ['Project Falcon ships on Friday. VG_FORGOTTEN_CANARY']

Observed output (pinned revision):

forget() returned:              0
recall after drain_writes():    ['Project Falcon ships on Friday. VG_FORGOTTEN_CANARY']

control A (sync remember -> forget):  forget returns 1; content is gone          # forget itself works
control B (remember_many -> reset):   recall returns []                          # reset() drains the queue

Expected Behavior

A destructive operation should observe the same write-queue discipline reset() already implements: forget() (and update()) should drain pending saves — or at least delete-then-drain-then-re-delete — before returning, so that a completed forget() call means the content can no longer appear in any later recall().

Suggested Fix

Add self.drain_writes() (and the _reset_lock mutual exclusion used by reset()) at the start of forget() and update(), mirroring the reset() implementation. Cost is bounded by the existing single-worker save pool.

Found during a correctness study of agent-memory lifecycle operations (delete/update vs. async derivation ordering). Happy to provide the full two-document schedule with per-step assertions.

Start in lib/crewai/memory/unified_memory.py, especially forget(), update(), reset(), reset_all(), recall(), and the remember_many() save path. Run the deterministic gated-embedder reproduction from the issue, then verify that destructive operations synchronize pending saves and that recall after drain_writes() cannot return forgotten content.