nervosnetwork / nervosnetwork/neuron

[Bug Report] Wallet synchronization stops permanently after a transient SQLITE_BUSY error

Open
#3,508 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

replied
Dominant language
TypeScript
Stars
204
Forks
95
Avg merge
8d 9h
Merged PRs (30d)
4

Description

Topic Type

Cannot Synchronize

Detail of the issue
Summary

Neuron's bundled CKB full node and indexer are fully synchronized, but the wallet synchronization UI remains permanently stuck at 70.32% after a transient SQLITE_BUSY: database is locked error.

The CKB RPC continues to work normally and reports no synchronization lag. However, Neuron stops updating its wallet cache after the SQLite error and does not recover until the wallet synchronization task is restarted.

Environment
  • OS: Windows
  • Neuron version: v0.209.0
  • Network: Mainnet
  • Node type: Internal full node
  • RPC URL: http://127.0.0.1:8114
Observed behavior
  1. The bundled node and indexer finished synchronizing with the mainnet tip.
  2. Neuron's wallet synchronization UI remained at 70.32%.
  3. main.log showed repeated transient SQLITE_BUSY errors while wallet synchronization was progressing.
  4. At 2026-08-30T06:08:25.245Z, the full synchronization loop logged Error connecting to Indexer: SQLITE_BUSY: database is locked.
  5. The final persisted wallet synchronization height was 14,278,433, and no later Database: saved synced block messages appeared.
  6. The UI continued displaying the last synchronization percentage instead of recovering or reporting that the wallet synchronization worker had stopped.

The problem is intermittent rather than deterministically reproducible. It appears during the initial high-write wallet synchronization phase.

Screenshot
Image
Full debug information

neuron_debug_1788170833775.zip

RPC verification

The following values were queried from the same RPC endpoint used by Neuron:

Port  Status   NodeTip   IndexerTip  BestKnown  IndexerLag  IBD    AssumeReached
8114  CKB RPC  20315476  20315476    20315476   0           False  True

This confirms that:

  • the node tip, indexer tip, and best-known tip were identical;
  • the indexer had zero-block lag;
  • the node was no longer in initial block download;
  • the configured assume-valid target had been reached.

The remaining 70.32% therefore represented Neuron's wallet cache progress, not CKB node or indexer progress.

Expected behavior
  • A transient SQLITE_BUSY error should be retried with backoff.
  • The wallet synchronization loop should continue with the next polling iteration.
  • If automatic recovery is impossible, Neuron should restart the synchronization worker or show an actionable error instead of leaving a stale percentage in the UI.
Actual behavior
  • The synchronization loop exits after one transient SQLite lock error.
  • The existing indexer-error worker-restart path is not reached.
  • The UI remains permanently stuck at the last reported percentage even though the bundled node and indexer are fully synchronized.
Source analysis

SQLITE_BUSY indicates contention on Neuron's wallet SQLite database: one connection held an incompatible database lock while another database operation waited for the configured timeout and then failed. It does not indicate CKB indexer lag or database corruption. The log message Error connecting to Indexer is therefore misleading in this case—the underlying exception came from SQLite.

The source links below are pinned to commit 484a46dadde2a0adfd7975d0159be898b199cf80, currently the tip of the rc/v0.209.0 branch and declaring package version 0.209.0. The relevant files were also compared with the current develop branch and are unchanged.

The current implementation has several relevant behaviors:

  1. SQLite is configured with a busy_timeout of only 3 seconds:
    packages/neuron-wallet/src/database/chain/ormconfig.ts
    No journal_mode=WAL configuration was found in the current database initialization code.
  2. Full wallet synchronization persists a fetched transaction, its inputs, previous outputs, outputs, and locks in an explicit multi-statement transaction. During the initial high-write synchronization phase, this can hold the SQLite write lock while data is saved in chunks:
    packages/neuron-wallet/src/services/tx/transaction-persistor.ts
  3. The transaction status listener runs every 5 seconds and can update or delete wallet transactions concurrently with synchronization. It rethrows database errors other than ConnectionNotFoundError, producing an unhandled rejection for SQLITE_BUSY:
    packages/neuron-wallet/src/block-sync-renderer/tx-status-listener.ts
  4. The directly observed failures occurred in the block-sync child process, where the full-sync persistence queue, indexer-cache updates, and transaction-status tracking can access the same database concurrently. The main process also opens the same chain database file, so cross-process access is an additional possible contender, but the current log does not show that the main process held the lock in this incident:
    packages/neuron-wallet/src/block-sync-renderer/task.ts and packages/neuron-wallet/src/controllers/networks/chain-info.ts
  5. Queue.start() already has a recovery path: if the awaited connector startup rejects, it sends indexer-error, and the parent process responds by resetting the synchronization task:
    packages/neuron-wallet/src/block-sync-renderer/sync/queue.ts and packages/neuron-wallet/src/block-sync-renderer/index.ts
  6. That recovery path is bypassed in this case. FullSynchronizer.connect() starts initSync() without awaiting or returning its promise, while initSync() catches the error itself and only logs it. Consequently, Queue.start() has already completed successfully and never receives a rejection, indexer-error is never sent, and the polling loop remains stopped:
    packages/neuron-wallet/src/block-sync-renderer/sync/full-synchronizer.ts
  7. By contrast, the transaction save queue retries its own failed operation every 2 seconds. This protects an in-flight task and may allow already cached transaction batches to keep draining, but it does not restart the dead initSync() polling loop, resume indexer-cache polling, or restore periodic progress emission:
    packages/neuron-wallet/src/block-sync-renderer/sync/queue.ts

The evidence therefore establishes that the failure was caused by contention on the wallet database and that the transient error terminated the synchronization polling loop without triggering the existing restart mechanism. The log does not include query-level or transaction begin/commit tracing, so it cannot identify the exact connection or transaction that held the lock at that instant. The most likely contenders are overlapping database operations inside the block-sync child process—particularly full-sync persistence/indexer-cache work and periodic transaction-status tracking—but this remains a source-based hypothesis rather than a proven lock-owner attribution. The status listener can be either a competing writer or the operation that receives SQLITE_BUSY, depending on timing.

The final Database: saved synced block #14278433 line does not prove that the retrying transaction-save queue recovered after initSync() stopped. That message is emitted by the main process when an asynchronously handled cache-tip-block-updated event is persisted:
packages/neuron-wallet/src/controllers/sync-api.ts and packages/neuron-wallet/src/models/synced-block-number.ts. One source-consistent explanation is that the last progress event had already been dispatched before the polling-loop error and completed asynchronously afterward. No later progress event was persisted.

This is not inherently Windows-specific because the competing SQLite access paths are cross-platform. Windows Defender, slower storage, or other host I/O may lengthen transaction time and make the race easier to trigger, but they are not required for the code path to fail.

Possible fix direction
  • Handle SQLITE_BUSY inside each synchronization iteration and retry with bounded exponential backoff.
  • Ensure a transient database error cannot terminate the lifetime of the full synchronization loop.
  • If retries are exhausted, explicitly observe the background initSync() failure and send indexer-error so the existing worker-reset path runs. Simply awaiting the long-lived loop before the event subscriptions are installed would prevent normal synchronization events from being consumed.
  • Avoid overlapping wallet-database writes between synchronization and periodic transaction-status tracking, or serialize those operations.
  • Consider whether WAL mode or a longer busy timeout is appropriate as additional protection.
  • Report the underlying database operation accurately instead of labeling every failure as an indexer connection error.
  • Add regression tests verifying that one synchronization iteration can recover from SQLITE_BUSY and that an exhausted retry path triggers the existing worker restart.
bundled-ckb.log

No relevant bundled CKB error. The bundled node and indexer were fully synchronized.

bundled-ckb-light-mainnet.log

No response — the issue occurred with the internal full node.

main.log

The complete main.log is included in the debug archive attached above.

Relevant log excerpt:

[2026-08-30T06:07:53.193Z] [info]  Database: saved synced block #14278352
[2026-08-30T06:08:05.973Z] [info]  Database: saved synced block #14278412
[2026-08-30T06:08:20.644Z] [warn]  status tracking error: QueryFailedError: SQLITE_BUSY: database is locked
[2026-08-30T06:08:20.646Z] [error] Unhandled Rejection in task: Reason: QueryFailedError: SQLITE_BUSY: database is locked
[2026-08-30T06:08:25.245Z] [error] Error connecting to Indexer: SQLITE_BUSY: database is locked
[2026-08-30T06:08:26.621Z] [info]  Database: saved synced block #14278433

No later Database: saved synced block entries appeared after the persisted wallet synchronization height 14,278,433.

Earlier in the same synchronization run, similar SQLITE_BUSY errors were also observed around wallet cache heights 11,858,994 and 14,230,686, but synchronization temporarily continued after those occurrences.

status.log

No response. Relevant RPC status is included above.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with packages/neuron-wallet/src/block-sync-renderer/sync/full-synchronizer.ts and sync/queue.ts to trace how SQLITE_BUSY errors leave the polling loop without triggering indexer-error. Then read tx-status-listener.ts and ormconfig.ts for concurrent database access and timeout behavior. Add regression coverage for recovery and worker restart, and verify wallet synchronization resumes or reports an actionable failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
sqlite, typescript
Domain
blockchain, database
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.