JOINs/reference fields broken on node restored from Raft snapshot

Summary

After following the official recovery procedure for a node that has been down for an extended period (documented at high-availability.html#recovery-after-an-extended-period), the restored node becomes healthy (/health returns ok, /debug shows state: 4) but all JOINs using reference fields fail with:

Failed to join on `product_vehicle_fitments_se`: No reference field found.

The collection schema metadata correctly shows the reference fields on the restored node, but the internal reference/JOIN index is not rebuilt from the Raft snapshot. This means the documented recovery procedure leaves the node in a state where it serves 200 OK for simple queries but returns errors for any query involving JOINs.

This was discovered while following the recovery steps recommended in #2821.

Environment

Typesense version	30.1
Cluster topology	3-node Raft
Memory per node	40–60 GB
Collections with references	4 (see below)
Total documents across refs	~51M

Collections with reference fields

Collection	Documents	Reference fields
product_vehicle_fitments_se	25,149,729	variant_pid → products_se.variant_pid, vehicle_id → vehicles_se.vehicle_id
product_vehicle_fitments_dk	25,149,543	variant_pid → products_dk.variant_pid, vehicle_id → vehicles_dk.vehicle_id
products_se	416,813	primary_level_3_category_id → categories_se.category_id
products_dk	310,406	primary_level_3_category_id → categories_dk.category_id

Recovery steps followed (exactly per docs)

1. Stopped Typesense on the unhealthy follower (Node 2).
2. Removed Node 2's IP from the nodes file on the two healthy nodes. Waited 30s for auto-refresh.
3. Verified the remaining 2-node cluster was healthy (leader + 1 follower, both /health → ok).
4. Triggered a snapshot on the leader via POST /operations/snapshot — confirmed success:

   snapshot_save_done, last_included_index=12134525 last_included_term=20
   

5. Cleared the Typesense data directory on Node 2 (rm -rf /data/*).
6. Added Node 2's IP back to the nodes file on all nodes.
7. Started Typesense on Node 2.

Node 2 loaded the fresh snapshot, caught up to committed index with zero lag, and reported healthy:

Term: 20, pending_queue: 0, last_index: 12134545, committed: 12134545,
  known_applied: 12134545, applying: 0, pending_writes: 0, queued_writes: 0

Docker health check also passed: Up 10 minutes (healthy).

The problem

Despite the node being fully healthy and caught up, all JOIN queries fail on this node only:

Query tested (same query, 3 nodes)

curl -s "http://<NODE>:8108/multi_search" \
  -H "X-TYPESENSE-API-KEY: $KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "searches": [{
      "collection": "vehicles_se",
      "q": "*",
      "query_by": "manufacturer",
      "per_page": 2,
      "filter_by": "$product_vehicle_fitments_se($products_se(product_pid:=2510966))",
      "include_fields": "vehicle_id,$product_vehicle_fitments_se(variant_pid,vehicle_id,$products_se(product_pid))"
    }]
  }'

Results

Node	Role	/health	/debug	JOIN query result
Node 1 (never went down)	follower	ok	state: 4	✅ 4504 hits
Node 2 (restored from snapshot)	follower	ok	state: 4	❌ Failed to join on product_vehicle_fitments_se: No reference field found.
Node 3 (leader)	leader	ok	state: 1	✅ 4504 hits

Schema on Node 2 shows reference fields correctly

{
  "name": "product_vehicle_fitments_se",
  "num_documents": 25149729,
  "fields": [
    {"name": "variant_pid", "reference": "products_se.variant_pid", ...},
    {"name": "vehicle_id", "reference": "vehicles_se.vehicle_id", ...},
    ...
  ]
}

The schema metadata is identical across all 3 nodes. The reference fields are declared and the document counts match. But the internal reference/JOIN index that Typesense uses to resolve $collection() filter syntax was not rebuilt from the snapshot.

Additional findings

• All 4 collections with reference fields are affected — both SE and DK markets.
• Collections without reference fields work fine on Node 2 (simple searches return correct results).
• Upserting all data into the existing collections on Node 2 does not fix the issue — the reference index is not rebuilt by upserts.
• The only known fix is to drop and recreate the collections (schema + data), which defeats the purpose of the snapshot recovery procedure.

Root cause hypothesis

The Raft snapshot preserves collection schema metadata (including reference field declarations) and document data, but does not preserve or rebuild the in-memory reference/JOIN index. When a node loads a snapshot, it reconstructs collections and indexes documents but skips the reference index construction step. This index is presumably only built during the create collection + import path, not during snapshot restoration.

Impact

Following the documented recovery procedure for extended outages (#2821) leaves the restored node serving broken JOIN queries. Since reads are served by whichever node receives the request, this means a percentage of all JOIN-dependent searches will fail with errors in production, even though the node appears fully healthy.

Expected behavior

A node restored from a Raft snapshot should have fully functional reference/JOIN indexes, identical behavior to the other nodes in the cluster.

Workaround

The only current workaround is to drop and recreate all collections with reference fields, then re-import all data. For large datasets (50M+ documents in our case) this means significant search downtime.

---

Related: #2821 — the original deadlock issue that led to the recovery procedure documented in the HA guide. The recovery steps work for getting the node healthy, but leave JOINs broken.

Start with the Raft snapshot restoration path and the initialization of reference/JOIN indexes, using the recovery steps and the snapshot operation described here. Reproduce the issue with the supplied multi_search request on a restored node, then verify that JOIN queries work there as they do on the leader and unaffected follower.