typesense / typesense/typesense

Single node typesense instance on docker fails to restart

Open
#2,257 3 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
26.6k
Forks
973
Avg merge
18h 45m
Merged PRs (30d)
4

Description

I have a single node Typesense instance running on docker (following the getting started docs) that I inserted a modest number of documents to. After a restart though, this instance fails to start. It takes a minute or so to load all the previously indexed docs and then abruptly quits.

docker run -p 8108:8108 -v$(pwd)/typesense-data:/data typesense/typesense:28.0 \
  --data-dir /data --api-key=$TYPESENSE_API_KEY --enable-cors

Relevant part of the logs

I20250316 08:10:06.572754   556 collection_manager.cpp:425] Loaded 3 collection(s) so far
I20250316 08:10:06.619529   534 collection_manager.cpp:509] Loaded 3 collection(s).
I20250316 08:10:06.624069   534 raft_server.cpp:596] Finished loading collections from disk.
I20250316 08:10:06.635862   534 raft_server.cpp:607] Loaded 0conversation model(s).
I20250316 08:10:06.636823   534 raft_server.cpp:611] Initializing batched indexer from snapshot state...
I20250316 08:10:06.643399   534 batched_indexer.cpp:595] req_res.start_ts: 1742110003399726, req_res.next_chunk_index: 1
I20250316 08:10:06.646940   534 batched_indexer.cpp:624] Restored 1 in-flight requests from snapshot.
I20250316 08:10:06.658744   534 raft_server.cpp:624] Loaded 0 personalization model(s).
I20250316 08:10:06.664312   534 raft_server.h:294] Configuration of this group is 172.17.0.2:8107:8108
I20250316 08:10:06.699020   534 snapshot_executor.cpp:264] node default_group:172.17.0.2:8107:8108 snapshot_load_done, last_included_index: 4533 last_included_term: 2 peers: "172.17.0.2:8107:8108"
I20250316 08:10:06.750931   406 raft_meta.cpp:521] Loaded single stable meta, path /data/state/meta term 4 votedfor 172.17.0.2:8107:8108 time: 22506
I20250316 08:10:06.787732   406 node.cpp:608] node default_group:172.17.0.2:8107:8108 init, term: 4 last_log_id: (index=4546,term=4) conf: 172.17.0.2:8107:8108 old_conf:
I20250316 08:10:06.836714   406 node.cpp:1645] node default_group:172.17.0.2:8107:8108 term 4 start vote and grant vote self
I20250316 08:10:06.987057   406 raft_meta.cpp:546] Saved single stable meta, path /data/state/meta term 5 votedfor 172.17.0.2:8107:8108 time: 78848
I20250316 08:10:07.079692   406 node.cpp:1899] node default_group:172.17.0.2:8107:8108 term 5 become leader of group 172.17.0.2:8107:8108
I20250316 08:10:07.137110   406 raft_server.cpp:136] Node last_index: 4546
I20250316 08:10:07.140154   406 typesense_server_utils.cpp:298] Typesense peering service is running on 172.17.0.2:8107
I20250316 08:10:07.140748   406 typesense_server_utils.cpp:299] Snapshot interval configured as: 3600s
I20250316 08:10:07.141906   406 typesense_server_utils.cpp:300] Snapshot max byte count configured as: 4194304
W20250316 08:10:07.142136   406 controller.cpp:1550] SIGINT was installed with 1
I20250316 08:10:07.154639   406 raft_server.cpp:683] Term: 5, pending_queue: 0, last_index: 4547, committed: 4547, known_applied: 4533, applying: 0, pending_writes: 0, queued_writes: 1, local_sequence: 2089607
W20250316 08:10:07.159222   406 node.cpp:843] [default_group:172.17.0.2:8107:8108 ] Refusing concurrent configuration changing
E20250316 08:10:07.165160   539 raft_server.h:62] Peer refresh failed, error: Doing another configuration change
I20250316 08:10:09.855855   534 raft_server.h:294] Configuration of this group is 172.17.0.2:8107:8108
I20250316 08:10:09.908322   534 raft_server.h:294] Configuration of this group is 172.17.0.2:8107:8108
I20250316 08:10:09.934266   534 raft_server.h:294] Configuration of this group is 172.17.0.2:8107:8108
I20250316 08:10:09.955137   534 node.cpp:3298] node default_group:172.17.0.2:8107:8108 reset ConfigurationCtx, new_peers: 172.17.0.2:8107:8108, old_peers: 172.17.0.2:8107:8108
I20250316 08:10:09.982980   534 raft_server.h:277] Node becomes leader, term: 5

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the restart with the provided Docker command and persisted /data volume, then inspect the startup path in collection_manager.cpp, raft_server.cpp, batched_indexer.cpp, snapshot_executor.cpp, raft_meta.cpp, and node.cpp. Use the logged configuration-change and peer-refresh messages as the starting point. Done means a single-node container restarts successfully and serves its previously indexed documents without exiting.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, docker
Domain
backend, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.