Split-brain / phantom documents: doc_id <-> seq_id store mapping desync (GET 404 but present in export, "Error while getting seq_id")

Bug Description

Documents can end up in a split-brain / "phantom" state where the seq_id store key (which holds the document JSON) and the doc_id store key (which maps id -> seq_id) get out of sync. When that happens:

• GET /collections/<c>/documents/<id> returns 404, and the document does not appear in search,
• yet the same document still appears when you stream a full GET /collections/<c>/documents/export,
• and an export with a non-empty include_fields/exclude_fields emits the line Error while getting seq_id of `<id>`: Not found. instead of the document JSON,
• num_documents drifts above the true number of unique document ids (orphan seq_id keys are re-indexed and counted as separate documents on reload).

The split is durable across restarts: on reload, Typesense scans the seq_id key family and re-indexes each entry but never rewrites/repairs the doc_id key, so once the two key families disagree they stay disagreed.

A common downstream symptom: if such a document is missing a field key that is non-optional in the schema, any schema PATCH that drops and re-adds a non-optional field re-validates every stored document and fails with:

Field `owner_id` has been declared in the schema, but is not found in the documents already present in the collection. If you still want to add this field, set it as `optional: true`.

even though the operator never asked to add a new field (they only re-declared an existing one, e.g. to fix a reference).

Reproduction Steps

The script below is self-contained (only docker + curl needed). It has two independent parts:

• PART 1 (deterministic, runs on 30.1 and 30.2): creates a stored document that legitimately lacks a key (declared optional), then re-declares that field as non-optional via drop+re-add and shows the strict full re-validation throwing the exact error above.
• PART 2 (timing-dependent): produces a durable doc_id <-> seq_id store desync by SIGKILL-ing the container mid bulk-delete and re-importing the same ids, then shows the export seq_id error line and/or num_documents count drift after a clean restart.

#!/bin/bash
# Phantom / split-brain document: doc_id <-> seq_id store mapping desync.
#
# WHAT A PHANTOM DOC IS:
#   (A) UNREACHABLE:   GET /collections/<c>/documents/<id> -> 404 AND not in search,
#                      yet the doc appears in a full /documents/export stream.
#   (B) MISSING KEYS:  the STORED doc JSON is missing field keys that are
#                      non-optional in the schema.
#   (C) EXPORT ERROR:  a full export (with include/exclude_fields set) emits
#                      `Error while getting seq_id of <id>: Not found.` instead of
#                      the doc JSON.
#
# ROOT-CAUSE MECHANISM (line refs are from the 30.x source tree):
#   Every doc is two rocksdb keys:
#     doc_id key:  "<cid>_$DI_<id>"        -> seq_id     (collection.cpp get_doc_id_key)
#     seq_id key:  "<cid>_seqid_<seq_id>"  -> doc JSON   (collection.cpp get_seq_id_key)
#   GET-by-id and export's per-doc lookup BOTH resolve id->seq_id first
#   (Collection::get ; doc_id_to_seq_id), but a full export SCANS THE seq_id KEY
#   FAMILY DIRECTLY (core_api.cpp). If the seq_id key exists but the doc_id key is
#   gone, the doc is in export but GET 404s (A) and export's doc_id_to_seq_id
#   returns "Not found" -> the exact (C) line is printed.
#   The two keys are NOT written/removed atomically as a single batch on every
#   path: remove_document issues two separate store->remove calls and
#   remove_if_found_many does the same per doc in a loop. The master runs with
#   WAL disabled (store.cpp), so durability is by async memtable flush; a hard
#   crash can leave the two keys in different flush states -> the split.
#   On reload, load_collection scans seq_id keys and re-indexes each one but
#   NEVER rewrites the doc_id key, so the split is durable across restarts and an
#   orphan seq_id key is even counted as a separate document.
#
# Requirements: docker, curl. (No Python/jq needed.)

set -u
export MSYS_NO_PATHCONV=1
export MSYS2_ARG_CONV_EXCL='*'

TYPESENSE_API_KEY=xyz
PART1_VERSIONS=${PART1_VERSIONS:-"30.1 30.2"}
PART2_VERSION=${PART2_VERSION:-30.2}
KILL_ATTEMPTS=${KILL_ATTEMPTS:-1}

PORT=${PORT:-8208}
H=http://localhost:${PORT}
CN=typesense-phantom-repro
VOL=ts-data-${CN}

PASS_PART1=0
PASS_PART2=0

require_cmd() { command -v "$1" >/dev/null 2>&1 || { echo "Required command not found: $1"; exit 127; }; }

api() { local m="$1" p="$2"; shift 2; curl -sS "${H}${p}" -X "${m}" -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" "$@"; }

wait_for_ready() {
  for _ in $(seq 1 180); do
    curl -s -m 3 "${H}/health" 2>/dev/null | grep -q '"ok":true' && return 0
    sleep 1
  done
  echo "Typesense did not become ready. Recent logs:"; docker logs --tail 30 "${CN}" 2>&1 || true
  return 1
}

# NOTE: //data (double leading slash) - on Git Bash / MSYS a single /data is
# rewritten to a Windows path and Typesense crashes. // is left untouched.
# On Linux/macOS //data is equivalent to /data, so this is portable.
start_fresh() {
  local version="$1"
  docker rm -f "${CN}" >/dev/null 2>&1 || true
  docker volume rm "${VOL}" >/dev/null 2>&1 || true
  docker volume create "${VOL}" >/dev/null
  docker run -d --name "${CN}" -p "${PORT}:8108" -v "${VOL}://data" \
    "typesense/typesense:${version}" \
    --data-dir //data --api-key="${TYPESENSE_API_KEY}" --enable-cors >/dev/null
  wait_for_ready
}

cleanup() {
  echo ""; echo "=== Cleanup ==="
  command -v docker >/dev/null 2>&1 || return 0
  docker rm -f "${CN}" >/dev/null 2>&1 || true
  docker volume rm "${VOL}" >/dev/null 2>&1 || true
}

require_cmd docker
require_cmd curl
trap cleanup EXIT

#############################################################################
# PART 1: missing-key stored doc + the strict-revalidation PATCH error
#############################################################################
for V in ${PART1_VERSIONS}; do
  echo ""
  echo "############################################################"
  echo "### PART 1 on Typesense ${V}: missing-key doc -> PATCH error"
  echo "############################################################"
  start_fresh "${V}" || exit 1

  # referenced collection behind an alias
  api POST "/collections" -H "Content-Type: application/json" -d '{
    "name":"ref_data",
    "fields":[{"name":"parent_id","type":"string","facet":true},
              {"name":"owner_id","type":"string","facet":true}]}' >/dev/null
  api PUT "/aliases/ref_alias" -H "Content-Type: application/json" -d '{"collection_name":"ref_data"}' >/dev/null
  api POST "/collections/ref_data/documents" -H "Content-Type: application/json" \
    -d '{"id":"p1","parent_id":"p1","owner_id":"o1"}' >/dev/null
  sleep 1

  # referencing collection: parent_id NON-optional ref,
  # owner_id declared OPTIONAL ref (so a doc may legally omit it).
  api POST "/collections" -H "Content-Type: application/json" -d '{
    "name":"items",
    "fields":[
      {"name":"parent_id","type":"string","facet":true,"sort":true,"reference":"ref_alias.parent_id","async_reference":true},
      {"name":"owner_id","type":"string","facet":true,"sort":true,"optional":true,"reference":"ref_alias.owner_id","async_reference":true},
      {"name":"label","type":"string"}]}' >/dev/null
  sleep 1

  # import a doc that OMITS owner_id (legal while the field is optional)
  api POST "/collections/items/documents" -H "Content-Type: application/json" \
    -d '{"id":"doc1","parent_id":"p1","label":"x"}' >/dev/null
  sleep 1

  echo "Stored doc1 (note: NO owner_id key present):"
  STORED=$(api GET "/collections/items/documents/export?exclude_fields=zzz")
  echo "  ${STORED}"
  if echo "${STORED}" | grep -q '"owner_id"'; then
    echo "  UNEXPECTED: owner_id present; missing-key state not set up."
  fi

  # Now re-declare owner_id as NON-optional via drop+re-add.
  # Typesense re-validates ALL existing docs; doc1 lacks the key -> error.
  echo ""
  echo "drop+re-add owner_id as NON-optional (strict full re-validation):"
  PATCH_BODY=$(api PATCH "/collections/items" -H "Content-Type: application/json" -w $'\n__HTTP=%{http_code}' -d '{
    "fields":[{"name":"owner_id","drop":true},
              {"name":"owner_id","type":"string","facet":true,"sort":true,"reference":"ref_alias.owner_id","async_reference":true}]}')
  PATCH_HTTP=$(echo "${PATCH_BODY}" | sed -n 's/^__HTTP=//p' | tail -1)
  PATCH_MSG=$(echo "${PATCH_BODY}" | sed '/^__HTTP=/d')
  echo "  HTTP ${PATCH_HTTP}: ${PATCH_MSG}"

  if [ "${PATCH_HTTP}" = "400" ] && echo "${PATCH_MSG}" | grep -q 'not found in the documents already present'; then
    echo "  PART 1 (${V}): PASS - strict re-validation error reproduced."
    PASS_PART1=$((PASS_PART1+1))
  else
    echo "  PART 1 (${V}): did not produce the expected error."
  fi
done

#############################################################################
# PART 2: durable store/index id-mapping desync via SIGKILL mid bulk-delete
#############################################################################
echo ""
echo "############################################################"
echo "### PART 2 on Typesense ${PART2_VERSION}: store/index desync"
echo "############################################################"
start_fresh "${PART2_VERSION}" || exit 1
api POST "/collections" -H "Content-Type: application/json" -d '{"name":"c","fields":[{"name":"v","type":"string"}]}' >/dev/null
seq 1 4000 | awk '{print "{\"id\":\"d"$1"\",\"v\":\"val"$1"\"}"}' \
  | api POST "/collections/c/documents/import?action=create" -H "Content-Type: text/plain" --data-binary @- >/dev/null
echo "Imported 4000 docs. num_documents=$(api GET '/collections/c' | grep -oE '\"num_documents\":[0-9]+')"

for attempt in $(seq 1 "${KILL_ATTEMPTS}"); do
  echo ""
  echo "=== kill attempt ${attempt}: SIGKILL mid bulk-delete, then re-import same ids ==="
  # delete (almost) everything with a small batch so many per-doc two-key removes
  # are in flight, and SIGKILL during it (no graceful flush).
  ( api DELETE "/collections/c/documents?filter_by=v:!=nope&batch_size=40" >/dev/null 2>&1 ) &
  DELPID=$!
  sleep 0.2
  docker kill "${CN}" >/dev/null 2>&1
  wait $DELPID 2>/dev/null

  docker start "${CN}" >/dev/null
  wait_for_ready || exit 1

  # re-import the same ids: any whose doc_id key was deleted are treated as NEW
  # docs and get fresh seq_ids, orphaning the surviving old seq_id keys.
  seq 1 4000 | awk '{print "{\"id\":\"d"$1"\",\"v\":\"val"$1"\"}"}' \
    | api POST "/collections/c/documents/import?action=upsert" -H "Content-Type: text/plain" --data-binary @- >/dev/null
done

# Clean restart, then measure the desync.
docker restart "${CN}" >/dev/null
wait_for_ready || exit 1

EXP=$(api GET "/collections/c/documents/export?exclude_fields=zzz")
DOC_LINES=$(echo "${EXP}" | grep -c '^{')
UNIQ_IDS=$(echo "${EXP}" | grep -oE '"id":"[^"]+"' | sort -u | wc -l | tr -d ' ')
NUMDOCS=$(api GET "/collections/c" | grep -oE '"num_documents":[0-9]+' | grep -oE '[0-9]+')
SEQERR=$(echo "${EXP}" | grep -c 'Error while getting seq_id of')

echo ""
echo "After kill + re-import + clean restart:"
echo "  export doc lines : ${DOC_LINES}"
echo "  unique doc ids   : ${UNIQ_IDS}"
echo "  num_documents    : ${NUMDOCS}"
echo "  seq_id err lines : ${SEQERR}"

if [ "${SEQERR}" -gt 0 ]; then
  echo "  PART 2: PASS - export emitted the exact (C) seq_id error line:"
  echo "${EXP}" | grep 'Error while getting seq_id of' | head -3 | sed 's/^/    /'
  PASS_PART2=1
elif [ "${DOC_LINES}" -gt "${UNIQ_IDS}" ] || [ "${NUMDOCS}" -gt "${UNIQ_IDS}" ]; then
  echo "  PART 2: PASS - durable store/index id-mapping DESYNC (orphan seq_id keys):"
  echo "    ${DOC_LINES} export doc lines vs ${UNIQ_IDS} unique ids; num_documents=${NUMDOCS}."
  echo "    Same bug family as the phantom (split id<->seq_id store mapping) + count drift."
  PASS_PART2=1
else
  echo "  PART 2: no desync observed this run (kill timing missed the flush boundary)."
  echo "    Re-run; raise KILL_ATTEMPTS to widen the window."
fi

#############################################################################
echo ""
echo "============================================================"
echo "SUMMARY"
echo "============================================================"
echo "PART 1 (missing-key doc -> strict-revalidation PATCH error): ${PASS_PART1}/$(echo ${PART1_VERSIONS} | wc -w) version(s) PASS"
echo "PART 2 (store/index id-mapping desync + count drift): $( [ ${PASS_PART2} -eq 1 ] && echo PASS || echo 'not this run' )"
if [ "${PASS_PART1}" -gt 0 ] && [ "${PASS_PART2}" -eq 1 ]; then
  echo "OVERALL: PASS"
  exit 0
elif [ "${PASS_PART1}" -gt 0 ]; then
  echo "OVERALL: PARTIAL (PART 1 deterministic; PART 2 is timing-dependent)"
  exit 0
else
  echo "OVERALL: FAIL"
  exit 2
fi

Expected vs Actual

Expected behavior

• A document is reachable by GET /documents/<id> if and only if it appears in /documents/export; the two views never disagree.
• num_documents equals the number of unique document ids.
• The two store keys for a document (doc_id and seq_id) are always created/removed atomically, so a crash cannot leave one without the other; reload repairs any disagreement.
• Re-declaring an existing field (drop + re-add) does not fail on documents that were valid under the previous schema.

Actual behavior

• A document can be present in export but return 404 on GET /documents/<id> and be absent from search (split id <-> seq_id mapping).
• A full export with include_fields/exclude_fields set prints Error while getting seq_id of `<id>`: Not found. for such documents.
• num_documents is inflated above the unique-id count (orphan seq_id keys are re-indexed and counted) and the split survives restarts.
• A non-optional field re-declaration fails with Field ... is not found in the documents already present in the collection ... set it as optional: true when any stored document is missing that key (including these phantom documents).

Environment

• Typesense version: 30.2 (PART 2). PART 1 reproduces on both 30.1 and 30.2.
• Operating system: reproduced via the official typesense/typesense Docker image (linux/amd64); host-OS independent.
• Client library & version: none — raw curl.

Schema / Configuration

PART 1 (reference + optional-to-non-optional promotion):

{
  "name": "items",
  "fields": [
    { "name": "parent_id", "type": "string", "reference": "ref_alias.parent_id", "async_reference": true },
    { "name": "owner_id", "type": "string", "optional": true, "reference": "ref_alias.owner_id", "async_reference": true },
    { "name": "label", "type": "string" }
  ]
}

PART 2 (minimal, single string field):

{ "name": "c", "fields": [ { "name": "v", "type": "string" } ] }

Additional Context

Mechanism. Each document is stored as two RocksDB keys: a doc_id key (id -> seq_id) and a seq_id key (seq_id -> document JSON). GET-by-id and export's per-document lookup both resolve id -> seq_id first, but a full export scans the seq_id key family directly. So whenever the seq_id key survives while the doc_id key is gone (or points elsewhere), the document shows up in export but GET 404s, and export's reverse lookup prints the Error while getting seq_id of <id>: Not found. line.

These two keys are not always written/removed as a single atomic batch: the per-document remove path issues two separate store->remove calls (one per key), and the master runs with the RocksDB WAL disabled, so durability is by asynchronous memtable flush. A hard crash (or, on a multi-node cluster, the Raft snapshot install / log-replay path) can therefore leave the two keys in different durable states. On reload, the loader scans the seq_id key family and re-indexes each entry but never rewrites the doc_id key, so the disagreement is permanent and an orphan seq_id entry is even re-indexed and counted as a separate document (the num_documents drift).

Scope / honest notes from reproduction.

• PART 1 deterministically reproduces the strict-revalidation PATCH error from a stored document that legitimately lacks an (optional) key, then has it promoted to non-optional. This is the most common operator-visible symptom.
• PART 2 reproduces a durable doc_id <-> seq_id desync (orphan seq_id keys, count drift, and the export seq_id error line) on a single node by SIGKILL mid-delete + re-import. It is timing-dependent; raise KILL_ATTEMPTS or re-run if a given run misses the flush boundary.
• On a single node, a tight kill mid-delete tends to keep both store->remove calls in the same memtable, so they are lost/flushed together. The exact "GET 404 but present in export" polarity observed in the wild is most readily produced by the multi-node Raft snapshot/log-replay install path; PART 2 demonstrates the same store/index desync class.

Suggested fixes for discussion.

1. Write/remove the doc_id and seq_id keys for a document in a single atomic RocksDB WriteBatch so a crash cannot split them.
2. On collection load, detect and repair seq_id entries whose doc_id key is missing or inconsistent (rewrite the doc_id key, or drop the orphan seq_id entry) instead of blindly re-indexing and counting it.
3. Make the strict full-document re-validation on field re-declaration skip / report rather than hard-fail on documents that predate the change, or surface which document ids are missing the key.

Start with the mentioned key-handling paths in collection.cpp, the export scan in core_api.cpp, persistence behavior in store.cpp, and reload handling in load_collection. Run the supplied Docker and curl reproduction, then trace the separate doc_id and seq_id operations across deletion, restart, and re-import. Done means the reproduction no longer produces GET/export disagreement, seq_id errors, or document-count drift after a hard kill and restart.