Recent easy corruption of nodes on latest/nightly/testing tag for sync & bucket corruption log for fresh sync

Open
#911 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Quiet
Tech stack
docker, postgresql, shell, sqlite

Research direction

Start with the reported failures and the backtrace locations in /wd/src/ledger/LedgerManagerImpl.cpp:550 and /wd/src/main/ApplicationImpl.cpp:805, then compare behavior across the nightly, latest, and testing images. Use the PostgreSQL and SQLite corruption messages to establish a reproducible sync failure; done means identifying the cause and confirming that a fresh sync completes without database or bucket corruption.

Written by the indexing model from the issue text.

Description

bug
What version are you using?

multiple of
stellar/quickstart:nightly
stellar/quickstart:latest
stellar/quickstart:testing
at the moment being "Id": "sha256:e7f05fac9ce971eccaabfd528297380c682f39f0d687c02bc7547d273a360145",

What did you do?

nodes broken , reworked multiple times with different tags

What did you expect to see?

working

What did you see instead?

always have postgres broken

2026-03-02T06:18:44.085 [default INFO] Assigning calculated value of 2 to FAILURE_SAFETY
current exception: std::runtime_error("Bucket directory is corrupt")
backtrace:
   0: stellar::printCurrentBacktrace()
             at /wd/src/util/Backtrace.cpp:27:22
   1: stellar::printBacktraceAndAbort()
             at /wd/src/main/main.cpp:140:5
   2: <unknown>
   3: <unknown>
   4: __cxa_throw
   5: loadLastKnownLedgerInternal
             at /wd/src/ledger/LedgerManagerImpl.cpp:550:9
   6: start
             at /wd/src/main/ApplicationImpl.cpp:805:21
   7: runApp
             at /wd/src/main/ApplicationUtils.cpp:117:10
   8: operator()
             at /wd/src/main/CommandLine.cpp:1638:20
      __invoke<(lambda at main/CommandLine.cpp:1576:9) &>
             at /usr/lib/llvm-20/bin/../include/c++/v1/__type_traits/invoke.h:179:25
      __call<(lambda at main/CommandLine.cpp:1576:9) &>
             at /usr/lib/llvm-20/bin/../include/c++/v1/__type_traits/invoke.h:243:12
      __invoke_r<int, (lambda at main/CommandLine.cpp:1576:9) &>
             at /usr/lib/llvm-20/bin/../include/c++/v1/__type_traits/invoke.h:273:10
      operator()
             at /usr/lib/llvm-20/bin/../include/c++/v1/__functional/function.h:167:12
      operator()
             at /usr/lib/llvm-20/bin/../include/c++/v1/__functional/function.h:319:10
   9: operator()
             at /usr/lib/llvm-20/bin/../include/c++/v1/__functional/function.h:436:12
      operator()
             at /usr/lib/llvm-20/bin/../include/c++/v1/__functional/function.h:995:10
      runWithHelp
             at /wd/src/main/CommandLine.cpp:479:12
  10: run
             at /wd/src/main/CommandLine.cpp:1570:12
2026-03-02T06:14:18.407 [default INFO] Assigning calculated value of 2 to FAILURE_SAFETY
current exception: std::runtime_error("sqlite3_statement_backend::loadOne: database disk image is malformed")
2026-03-02 06:15:19 UTC [120668-1607] stellar@horizon ERROR:  could not read block 31756 in file "base/16384/16401": read only 0 of 8192 bytes
2026-03-02 06:15:19 UTC [120668-1608] stellar@horizon STATEMENT:  SELECT COALESCE(MAX(sequence), 0) FROM history_ledgers

even re-sync with diff image tag still seeing this

Dominant language
Shell
Stars
222
Forks
236
Avg merge
1d 12h
Merged PRs (30d)
12

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from stellar/quickstart

All issues in stellar/quickstart

Similar issues

More Shell/Bash issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.