taosdata / taosdata/TDengine

after "alter database replica 3" , the transaction keeps in the redoAction stage

Open
#24,201 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
C
Stars
25.1k
Forks
5k
Avg merge
4d 59m
Merged PRs (30d)
7

Description

事务详情如下所示

taos> show transactions\G;
*************************** 1.row ***************************
              id: 93177
     create_time: 2023-12-22 20:08:55.710
           stage: redoAction
            oper: alter-db
              db: program_analyze
          stable: 
    failed_times: 29299
  last_exec_time: 2023-12-23 14:17:46.859
last_action_info: action:20 code:0x520(Vnode not exist) msgType:alter-replica numOfEps:1 inUse:0 ep:0-h2.taosdata.com:6030 
Query OK, 1 row(s) in set (0.001840s)

并且某个子节点一直报同步失败的错误,并出现core,无法正常启动加入集群

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/bin/taosd'.
Program terminated with signal 6, Aborted.
#0  0x00007f96690a0337 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install glibc-2.17-292.el7.x86_64 libgcc-4.8.5-39.el7.x86_64 libstdc++-4.8.5-39.el7.x86_64
(gdb) bt
#0  0x00007f96690a0337 in raise () from /lib64/libc.so.6
#1  0x00007f96690a1b68 in abort () from /lib64/libc.so.6
#2  0x00007f9669099156 in __assert_fail_base () from /lib64/libc.so.6
#3  0x00007f9669099202 in __assert_fail () from /lib64/libc.so.6
#4  0x000000000124b1d0 in taosAssertDebug (condition=false, file=0x19bf078 "/data/release/3220/community/source/libs/sync/src/syncRaftLog.c", line=107, format=0x19bf05c "assert info not provided") at /data/release/3220/community/source/util/src/tlog.c:845
#5  0x0000000000eb7680 in raftLogRestoreFromSnapshot (pLogStore=0x7f964817ab50, snapshotIndex=-1) at /data/release/3220/community/source/libs/sync/src/syncRaftLog.c:107
#6  0x0000000000ea26e2 in syncNodeLogStoreRestoreOnNeed (pNode=0x7f96481ab240) at /data/release/3220/community/source/libs/sync/src/syncMain.c:794
#7  0x0000000000ea390c in syncNodeOpen (pSyncInfo=0x7f9658ab6040, vnodeVersion=5) at /data/release/3220/community/source/libs/sync/src/syncMain.c:1030
#8  0x0000000000e9f520 in syncOpen (pSyncInfo=0x7f9658ab6040, vnodeVersion=5) at /data/release/3220/community/source/libs/sync/src/syncMain.c:63
#9  0x00000000007c10a6 in vnodeSyncOpen (pVnode=0x7f96480008c0, path=0x7f9658ab6b60 "/data/lib/taos/vnode/vnode18", vnodeVersion=5) at /data/release/3220/community/source/dnode/vnode/src/vnd/vnodeSync.c:698
#10 0x00000000007ab744 in vnodeOpen (path=0x7f9658ab77e0 "vnode/vnode18", diskPrimary=0, pTfs=0x5e06930, msgCb=..., force=false) at /data/release/3220/community/source/dnode/vnode/src/vnd/vnodeOpen.c:474
#11 0x00000000007986e7 in vmOpenVnodeInThread (param=0x5e738c0) at /data/release/3220/community/source/dnode/mgmt/mgmt_vnode/src/vmInt.c:310
#12 0x00007f966a06ae65 in start_thread () from /lib64/libpthread.so.0
#13 0x00007f966916888d in clone () from /lib64/libc.so.6
(gdb) 

image

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the failing transaction details and the restore path in libs/sync/src/syncRaftLog.c:107, then trace the callers in syncMain.c, vnodeSync.c, and vnodeOpen.c shown in the backtrace. Reproduce or inspect the alter-replica failure and snapshot restore behavior. Done means the transaction leaves redoAction and the affected node can rejoin and start without the assertion crash.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, sql
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.