pingcap / pingcap/tiflash

tiflash crashes in `TiDB::decodeUtf8Char` with randgen test

Open
#9,491 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

component/compute impact/crash may-affects-5.4 may-affects-6.1 may-affects-6.5 may-affects-7.1 may-affects-7.5 may-affects-8.1 randgen severity/moderate type/bug
Dominant language
C++
Stars
1k
Forks
423
Avg merge
1d 15h
Merged PRs (30d)
24

Description

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)
# checkout the "unifytestframework-vector "branch of schrddl
https://github.com/PingCAP-QE/schrddl/tree/unifytestframework-vector 
# deploy the cluster
# run the randgen test
go build && ./schrddl  --concurrency=10
2. What did you expect to see? (Required)
3. What did you see instead (Required)
[2024/09/29 14:30:34.375 +08:00] [ERROR] [BaseDaemon.cpp:560] ["
  0x55bb0ee1f89a  faultSignalHandler(int, siginfo_t*, void*) [tiflash+211695770]
                  libs/libdaemon/src/BaseDaemon.cpp:211
  0x7f89ce79fd90  <unknown symbol> [libc.so.6+347536]
  0x55bb10b0a007  TiDB::decodeUtf8Char(char const*, unsigned long&) [tiflash+242016263]
    dbms/src/TiDB/Collation/Collator.cpp:48
  0x55bb10b0cbbd  TiDB::UCACICollator<TiDB::Unicode0400, true>::decodeChar(char const*, unsigned long&) [tiflash+242027453]
                  dbms/src/TiDB/Collation/Collator.cpp:460
  0x55bb10b0cb4b  TiDB::UCACICollator<TiDB::Unicode0400, true>::weight(unsigned long&, unsigned long&, unsigned long&, unsigned long, char const*) [tiflash+242027339]
                  dbms/src/TiDB/Collation/Collator.cpp:482
  0x55bb10b0c612  TiDB::UCACICollator<TiDB::Unicode0400, true>::compare(char const*, unsigned long, char const*, unsigned long) const [tiflash+242026002]
                  dbms/src/TiDB/Collation/Collator.cpp:392
  0x55bb101c82d4  void std::__1::__insertion_sort[abi:ue170006]<std::__1::_ClassicAlgPolicy, DB::ColumnString::LessWithCollation<true, TiDB::ITiDBCollator>&, DB::PODArray<unsigned long, 4096ul, Allocator<false>, 15ul, 16ul>::iterator>(DB::PODArray<unsigned long, 4096ul, Allocator<false>, 15ul, 16ul>::iterator, DB::PODArray<unsigned long, 4096ul, Allocator<false>, 15ul, 16ul>::iterator, DB::ColumnString::LessWithCollation<true, TiDB::ITiDBCollator>&) [tiflash+232309460]
                  dbms/src/Columns/ColumnString.cpp:358
  0x55bb101c74d2  void std::__1::__introsort<std::__1::_ClassicAlgPolicy, DB::ColumnString::LessWithCollation<true, TiDB::ITiDBCollator>&, DB::PODArray<unsigned long, 4096ul, Allocator<false>, 15ul, 16ul>::iterator, false>(DB::PODArray<unsigned long, 4096ul, Allocator<false>, 15ul, 16ul>::iterator, DB::PODArray<unsigned long, 4096ul, Allocator<false>, 15ul, 16ul>::iterator, DB::ColumnString::LessWithCollation<true, TiDB::ITiDBCollator>&, std::__1::iterator_traits<DB::PODArray<unsigned long, 4096ul, Allocator<false>, 15ul, 16ul>::iterator>::difference_type, bool) [tiflash+232305874]
    /DATA/disk1/ra_common/tiflash-env-17/sysroot/bin/../include/c++/v1/__algorithm/sort.h:757
  0x55bb101c7df0  void std::__1::__introsort<std::__1::_ClassicAlgPolicy, DB::ColumnString::LessWithCollation<true, TiDB::ITiDBCollator>&, DB::PODArray<unsigned long, 4096ul, Allocator<false>, 15ul, 16ul>::iterator, false>(DB::PODArray<unsigned long, 4096ul, Allocator<false>, 15ul, 16ul>::iterator, DB::PODArray<unsigned long, 4096ul, Allocator<false>, 15ul, 16ul>::iterator, DB::ColumnString::LessWithCollation<true, TiDB::ITiDBCollator>&, std::__1::iterator_traits<DB::PODArray<unsigned long, 4096ul, Allocator<false>, 15ul, 16ul>::iterator>::difference_type, bool) [tiflash+232308208]
                  /DATA/disk1/ra_common/tiflash-env-17/sysroot/bin/../include/c++/v1/__algorithm/sort.h:822
  0x55bb101c6f2b  void std::__1::__sort_dispatch[abi:ue170006]<std::__1::_ClassicAlgPolicy, DB::PODArray<unsigned long, 4096ul, Allocator<false>, 15ul, 16ul>::iterator, DB::ColumnString::LessWithCollation<true, TiDB::ITiDBCollator>>(DB::PODArray<unsigned long, 4096ul, Allocator<false>, 15ul, 16ul>::iterator, DB::PODArray<unsigned long, 4096ul, Allocator<false>, 15ul, 16ul>::iterator, DB::ColumnString::LessWithCollation<true, TiDB::ITiDBCollator>&) [tiflash+232304427]
                  /DATA/disk1/ra_common/tiflash-env-17/sysroot/bin/../include/c++/v1/__algorithm/sort.h:878
  0x55bb101c6e8c  void std::__1::__sort_impl[abi:ue170006]<std::__1::_ClassicAlgPolicy, DB::PODArray<unsigned long, 4096ul, Allocator<false>, 15ul, 16ul>::iterator, DB::ColumnString::LessWithCollation<true, TiDB::ITiDBCollator>>(DB::PODArray<unsigned long, 4096ul, Allocator<false>, 15ul, 16ul>::iterator, DB::PODArray<unsigned long, 4096ul, Allocator<false>, 15ul, 16ul>::iterator, DB::ColumnString::LessWithCollation<true, TiDB::ITiDBCollator>&) [tiflash+232304268]
                  /DATA/disk1/ra_common/tiflash-env-17/sysroot/bin/../include/c++/v1/__algorithm/sort.h:946
  0x55bb101b7e69  void std::__1::sort[abi:ue170006]<DB::PODArray<unsigned long, 4096ul, Allocator<false>, 15ul, 16ul>::iterator, DB::ColumnString::LessWithCollation<true, TiDB::ITiDBCollator>>(DB::PODArray<unsigned long, 4096ul, Allocator<false>, 15ul, 16ul>::iterator, DB::PODArray<unsigned long, 4096ul, Allocator<false>, 15ul, 16ul>::iterator, DB::ColumnString::LessWithCollation<true, TiDB::ITiDBCollator>) [tiflash+232242793]
                  /DATA/disk1/ra_common/tiflash-env-17/sysroot/bin/../include/c++/v1/__algorithm/sort.h:954
  0x55bb0ff0553a  void DB::ColumnString::LessWithCollation<false, void>::getPermutationWithCollationImpl<TiDB::ITiDBCollator>(DB::ColumnString const&, TiDB::ITiDBCollator const&, bool, unsigned long, DB::PODArray<unsigned long, 4096ul, Allocator<false>, 15ul, 16ul>&) [tiflash+229414202]
                  dbms/src/Columns/ColumnString.cpp:426
  0x55bb0feff515  DB::ColumnString::getPermutationWithCollationImpl(TiDB::ITiDBCollator const&, bool, unsigned long, DB::PODArray<unsigned long, 4096ul, Allocator<false>, 15ul, 16ul>&) const [tiflash+229389589]
                  dbms/src/Columns/ColumnString.cpp:458
  0x55bb0ff0bda0  DB::ColumnString::getPermutation(TiDB::ITiDBCollator const&, bool, unsigned long, int, DB::PODArray<unsigned long, 4096ul, Allocator<false>, 15ul, 16ul>&) const [tiflash+229440928]
                  dbms/src/Columns/ColumnString.h:329
  0x55bb0fef4ad3  DB::ColumnNullable::getPermutation(TiDB::ITiDBCollator const&, bool, unsigned long, int, DB::PODArray<unsigned long, 4096ul, Allocator<false>, 15ul, 16ul>&) const [tiflash+229346003]
                  dbms/src/Columns/ColumnNullable.cpp:415
  0x55bb0fc6c45e  DB::sortBlock(DB::Block&, std::__1::vector<DB::SortColumnDescription, std::__1::allocator<DB::SortColumnDescription>> const&, unsigned long) [tiflash+226690142]
                  dbms/src/Interpreters/sortBlock.cpp:416
  0x55bb1119aece  DB::PartialSortTransformOp::transformImpl(DB::Block&) [tiflash+248901326]
                  dbms/src/Operators/PartialSortTransformOp.cpp:22
  0x55bb110f9a0f  DB::TransformOp::transform(DB::Block&) [tiflash+248240655]
                  dbms/src/Operators/Operator.cpp:129
  0x55bb110f78d8  DB::PipelineExec::executeImpl() [tiflash+248232152]
                  dbms/src/Flash/Pipeline/Exec/PipelineExec.cpp:141
  0x55bb110f4cd5  DB::PipelineExec::execute() [tiflash+248220885]
                  dbms/src/Flash/Pipeline/Exec/PipelineExec.cpp:109
  0x55bb1111046f  DB::PipelineTaskBase::runExecute() [tiflash+248333423]
                  dbms/src/Flash/Pipeline/Schedule/Tasks/Impls/PipelineTaskBase.h:76
  0x55bb111102bc  DB::PipelineTask::executeImpl() [tiflash+248332988]
                  dbms/src/Flash/Pipeline/Schedule/Tasks/Impls/PipelineTask.h:38
  0x55bb1111f6a5  DB::Task::execute() [tiflash+248395429]
                  dbms/src/Flash/Pipeline/Schedule/Tasks/Task.cpp:128
  0x55bb1112ad2d  DB::CPUImpl::exec(std::__1::unique_ptr<DB::Task, std::__1::default_delete<DB::Task>>&) [tiflash+248442157]
                  dbms/src/Flash/Pipeline/Schedule/ThreadPool/TaskThreadPoolImpl.h:32
  0x55bb1112aaa7  DB::TaskThreadPool<DB::CPUImpl>::handleTask(std::__1::unique_ptr<DB::Task, std::__1::default_delete<DB::Task>>&) [tiflash+248441511]
                  dbms/src/Flash/Pipeline/Schedule/ThreadPool/TaskThreadPool.cpp:104
  0x55bb1112a6fe  DB::TaskThreadPool<DB::CPUImpl>::doLoop(unsigned long) [tiflash+248440574]
          dbms/src/Flash/Pipeline/Schedule/ThreadPool/TaskThreadPool.cpp:83
  0x55bb11129f21  DB::TaskThreadPool<DB::CPUImpl>::loop(unsigned long) [tiflash+248438561]
                  dbms/src/Flash/Pipeline/Schedule/ThreadPool/TaskThreadPool.cpp:62
  0x55bb1112c674  decltype(*std::declval<DB::TaskThreadPool<DB::CPUImpl>*>().*std::declval<void (DB::TaskThreadPool<DB::CPUImpl>::*)(unsigned long)>()(std::declval<unsigned long>())) std::__1::__invoke[abi:ue170006]<void (DB::TaskThreadPool<DB::CPUImpl>::*)(unsigned long), DB::TaskThreadPool<DB::CPUImpl>*, unsigned long, void>(void (DB::TaskThreadPool<DB::CPUImpl>::*&&)(unsigned long), DB::TaskThreadPool<DB::CPUImpl>*&&, unsigned long&&) [tiflash+248448628]
            /DATA/disk1/ra_common/tiflash-env-17/sysroot/bin/../include/c++/v1/__type_traits/invoke.h:308
  0x55bb1112c5ef  void std::__1::__thread_execute[abi:ue170006]<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void (DB::TaskThreadPool<DB::CPUImpl>::*)(unsigned long), DB::TaskThreadPool<DB::CPUImpl>*, unsigned long, 2ul, 3ul>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void (DB::TaskThreadPool<DB::CPUImpl>::*)(unsigned long), DB::TaskThreadPool<DB::CPUImpl>*, unsigned long>&, std::__1::__tuple_indices<2ul, 3ul>) [tiflash+248448495]
                  /DATA/disk1/ra_common/tiflash-env-17/sysroot/bin/../include/c++/v1/__thread/thread.h:221
  0x55bb1112c3a2  void* std::__1::__thread_proxy[abi:ue170006]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void (DB::TaskThreadPool<DB::CPUImpl>::*)(unsigned long), DB::TaskThreadPool<DB::CPUImpl>*, unsigned long>>(void*) [tiflash+248447906]
                  /DATA/disk1/ra_common/tiflash-env-17/sysroot/bin/../include/c++/v1/__thread/thread.h:232
  0x7f89ce7ea802  start_thread [libc.so.6+653314]"] [source=BaseDaemon] [thread_id=1748]
4. What is your TiFlash version? (Required)

The "feature/vector-index" branch

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The stack trace points to dbms/src/TiDB/Collation/Collator.cpp at decodeUtf8Char during sorting in dbms/src/Columns/ColumnString.cpp. Reproduce from the schrddl unifytestframework-vector branch with go build && ./schrddl --concurrency=10, then isolate the failing UTF-8 or collation input; done means the randgen workload no longer crashes and the failure has regression coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.