pingcap / pingcap/tiflash

The error and info log is printed in one line

Open
#6,887 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

component/storage may-affects-4.0 may-affects-5.0 may-affects-5.1 may-affects-5.2 may-affects-5.3 may-affects-5.4 may-affects-6.1 may-affects-6.5 severity/minor type/bug
Dominant language
C++
Stars
1k
Forks
423
Avg merge
1d 15h
Merged PRs (30d)
24

Description

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

Tiflash oom and the log is as follows. The error and info log is printed in one line and it troubles with the log search.

[2023/02/22 15:19:42.619 +08:00] [ERROR] [SegmentReader.cpp:123] ["ErrMsg: Memory limit (total) exceeded caused by 'RSS(Resident Set Size) much larger than limit' : process memory size would be 47.89 GiB for (attempt to allocate chunk of 1049120 bytes), limit of memory for data computing : 28.63 GiB StackTrace \n       0x1a03fb1\tDB::TiFlashException::TiFlashException(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::TiFlashError const&) [tiflash+27279281]\n                \tdbms/src/Common/TiFlashException.h:250\n       0x1a03543\tMemoryTracker::alloc(long, bool) [tiflash+27276611]\n                \tdbms/src/Common/MemoryTracker.cpp:152\n       0x1a03205\tMemoryTracker::alloc(long, bool) [tiflash+27275781]\n                \tdbms/src/Common/MemoryTracker.cpp:163\n       0x1a03205\tMemoryTracker::alloc(long, bool) [tiflash+27275781]\n                \tdbms/src/Common/MemoryTracker.cpp:163\n       0x1a1326b\tAllocator<false>::alloc(unsigned long, unsigned long) [tiflash+27341419]\n                \tdbms/src/Common/Allocator.cpp:68\n       0x72378f5\tDB::createReadBufferFromFileBaseByFileProvider(std::__1::shared_ptr<DB::FileProvider> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::EncryptionPath const&, unsigned long, std::__1::shared_ptr<DB::ReadLimiter> const&, DB::ChecksumAlgo, unsigned long, int) [tiflash+119765237]\n                \tdbms/src/Encryption/createReadBufferFromFileBaseByFileProvider.cpp:86\n       0x1962848\tDB::CompressedReadBufferFromFileProvider<false>::CompressedReadBufferFromFileProvider(std::__1::shared_ptr<DB::FileProvider>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::EncryptionPath const&, unsigned long, std::__1::shared_ptr<DB::ReadLimiter> const&, DB::ChecksumAlgo, unsigned long) [tiflash+26617928]\n                \tdbms/src/Encryption/CompressedReadBufferFromFileProvider.cpp:78\n       0x7a068d8\tDB::DM::DMFileReader::Stream::Stream(DB::DM::DMFileReader&, long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned long, unsigned long, std::__1::shared_ptr<DB::Logger> const&, std::__1::shared_ptr<DB::ReadLimiter> const&) [tiflash+127953112]\n                \tdbms/src/Storages/DeltaMerge/File/DMFileReader.cpp:158\n       0x7a0ddd4\tstd::__1::__function::__func<DB::DM::DMFileReader::DMFileReader(std::__1::shared_ptr<DB::DM::DMFile> const&, std::__1::vector<DB::DM::ColumnDefine, std::__1::allocator<DB::DM::ColumnDefine> > const&, bool, bool, bool, bool, unsigned long, DB::DM::DMFilePackFilter&&, std::__1::shared_ptr<DB::MarkCache> const&, bool, std::__1::shared_ptr<DB::DM::ColumnCache> const&, unsigned long, unsigned long, std::__1::shared_ptr<DB::FileProvider> const&, std::__1::shared_ptr<DB::ReadLimiter> const&, unsigned long, bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, bool, std::__1::shared_ptr<DB::DM::ScanContext> const&)::$_2, std::__1::allocator<DB::DM::DMFileReader::DMFileReader(std::__1::shared_ptr<DB::DM::DMFile> const&, std::__1::vector<DB::DM::ColumnDefine, std::__1::allocator<DB::DM::ColumnDefine> > const&, bool, bool, bool, bool, unsigned long, DB::DM::DMFilePackFilter&&, std::__1::shared_ptr<DB::MarkCache> const&, bool, std::__1::shared_ptr<DB::DM::ColumnCache> const&, unsigned long, unsigned long, std::__1::shared_ptr<DB::FileProvider> const&, std::__1::shared_ptr<DB::ReadLimiter> const&, unsigned long, bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, bool, std::__1::shared_ptr<DB::DM::ScanContext> const&)::$_2>, void (std::__1::vector<DB::IDataType::Substream, std::__1::allocator<DB::IDataType::Substream> > const&)>::operator()(std::__1::vector<DB::IDataType::Substream, std::__1::allocator<DB::IDataType::Substream> > const&) [tiflash+127983060]\n                \t/usr/local/bin/../include/c++/v1/__functional/function.h:345\n       0x7a077f0\tDB::DM::DMFileReader::DMF[2023/02/22 15:19:47.639 +08:00] [INFO] [BaseDaemon.cpp:1160] ["Welcome to TiFlash"] [thread_id=1]
[2023/02/22 15:19:47.639 +08:00] [INFO] [BaseDaemon.cpp:1161] ["Starting daemon with revision 54381"] [thread_id=1]
[2023/02/22 15:19:47.639 +08:00] [INFO] [BaseDaemon.cpp:1164] ["TiFlash build info: TiFlash\nRelease Version: v6.7.0-alpha\nEdition:         Community\nGit Commit Hash: 2d5909edc07402992ba1ff7c81f7aafeb227ccd3\nGit Branch:      heads/refs/tags/v6.7.0-alpha\nUTC Build Time:  2023-02-21 11:35:50\nEnable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto\nProfile:         RELWITHDEBINFO\n"] [thread_id=1]

image

2. What did you expect to see? (Required)
3. What did you see instead (Required)
4. What is your TiFlash version? (Required)

master

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by examining the log output around SegmentReader.cpp:123 and the startup entries in BaseDaemon.cpp:1160-1164. Reproduce the TiFlash OOM scenario and trace how the error and subsequent INFO records are emitted. Done means separate log records remain separately searchable while preserving the error stack trace.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.