pingcap / pingcap/tiflash

translation from `sum` ast node to tipb::expr has incorrect field type

Open
#5,365 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

component/compute may-affects-4.0 may-affects-5.0 may-affects-5.1 may-affects-5.2 may-affects-5.3 may-affects-5.4 may-affects-6.0 may-affects-6.1 severity/moderate type/bug
Dominant language
C++
Stars
1k
Forks
423
Avg merge
1d 15h
Merged PRs (30d)
24

Description

Bug Report

Please answer these questions before submitting your issue. Thanks!

Current transformation from sum aggregation ast node to tipb::expr has some incorrect behaviors.

  1. output type must be unsigned
  2. output value must not be null, however sum(null) should be null

And the cause might lies in Aggregation::toTiPBExecutor in dbms/src/Debug/astToExecutor.cpp:

        if (agg_sig == tipb::ExprType::Count || agg_sig == tipb::ExprType::Sum)
        {
            auto * ft = agg_func->mutable_field_type();
            ft->set_tp(TiDB::TypeLongLong);
            ft->set_flag(TiDB::ColumnFlagUnsigned | TiDB::ColumnFlagNotNull);
        }
1. Minimal reproduce step (Required)

By writing a simple test like

TEST_F(JoinExecutorTestRunner, agg)
try
{
    context.addMockTable("tmp", "t", {{"a", TiDB::TP::TypeLong}}, {toNullableVec<Int32>({1, {}})});
    auto request = context.scan("tmp", "t")
        .aggregation({Sum(col("a"))}, {col("a")})
        .build(context);

    {
        ASSERT_COLUMNS_EQ_R(createColumns({toNullableVec<Int64>({1, {}}), toNullableVec<Int64>({1, {}})}), executeStreams(request));
    }
}

And execute ./gtests_dbms --gtest_filter="JoinExecutorTestRunner.agg"

2. What did you expect to see? (Required)

Test passed.

3. What did you see instead (Required)
Code: 349. DB::Exception: Cannot convert NULL value to non-Nullable type

Stack trace:

     0x110988410        StackTrace::StackTrace() [gtests_dbms+4494296080]
                        dbms/src/Common/StackTrace.cpp:23
     0x11098844c        StackTrace::StackTrace() [gtests_dbms+4494296140]
                        dbms/src/Common/StackTrace.cpp:22
     0x104b788c8        DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) [gtests_dbms+4295002312]
                        dbms/src/Common/Exception.h:41
     0x104b77a20        DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) [gtests_dbms+4294998560]
                        dbms/src/Common/Exception.h:43
     0x1073fadcc        DB::FunctionCast::prepare(std::__1::shared_ptr<DB::IDataType const> const&, std::__1::shared_ptr<DB::IDataType const> const&) const::'lambda1'(DB::Block&, std::__1::vector<unsigned long, std::__1::allocator<unsigned long> > const&, unsigned long)::operator()(DB::Block&, std::__1::vector<unsigned long, std::__1::allocator<unsigned long> > const&, unsigned long) const [gtests_dbms+4337479116]
                        dbms/src/Functions/FunctionsConversion.h:2884
     0x1073fabb4        decltype(std::__1::forward<DB::FunctionCast::prepare(std::__1::shared_ptr<DB::IDataType const> const&, std::__1::shared_ptr<DB::IDataType const> const&) const::'lambda1'(DB::Block&, std::__1::vector<unsigned long, std::__1::allocator<unsigned long> > const&, unsigned long)&>(fp)(std::__1::forward<DB::Block&>(fp0), std::__1::forward<std::__1::vector<unsigned long, std::__1::allocator<unsigned long> > const&>(fp0), std::__1::forward<unsigned long>(fp0))) std::__1::__invoke<DB::FunctionCast::prepare(std::__1::shared_ptr<DB::IDataType const> const&, std::__1::shared_ptr<DB::IDataType const> const&) const::'lambda1'(DB::Block&, std::__1::vector<unsigned long, std::__1::allocator<unsigned long> > const&, unsigned long)&, DB::Block&, std::__1::vector<unsigned long, std::__1::allocator<unsigned long> > const&, unsigned long>(DB::FunctionCast::prepare(std::__1::shared_ptr<DB::IDataType const> const&, std::__1::shared_ptr<DB::IDataType const> const&) const::'lambda1'(DB::Block&, std::__1::vector<unsigned long, std::__1::allocator<unsigned long> > const&, unsigned long)&, DB::Block&, std::__1::vector<unsigned long, std::__1::allocator<unsigned long> > const&, unsigned long&&) [gtests_dbms+4337478580]
                        /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.1.sdk/usr/include/c++/v1/type_traits:3694
     0x1073fab24        void std::__1::__invoke_void_return_wrapper<void, true>::__call<DB::FunctionCast::prepare(std::__1::shared_ptr<DB::IDataType const> const&, std::__1::shared_ptr<DB::IDataType const> const&) const::'lambda1'(DB::Block&, std::__1::vector<unsigned long, std::__1::allocator<unsigned long> > const&, unsigned long)&, DB::Block&, std::__1::vector<unsigned long, std::__1::allocator<unsigned long> > const&, unsigned long>(DB::FunctionCast::prepare(std::__1::shared_ptr<DB::IDataType const> const&, std::__1::shared_ptr<DB::IDataType const> const&) const::'lambda1'(DB::Block&, std::__1::vector<unsigned long, std::__1::allocator<unsigned long> > const&, unsigned long)&, DB::Block&, std::__1::vector<unsigned long, std::__1::allocator<unsigned long> > const&, unsigned long&&) [gtests_dbms+4337478436]
                        /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.1.sdk/usr/include/c++/v1/__functional_base:348
     0x1073faabc        std::__1::__function::__alloc_func<DB::FunctionCast::prepare(std::__1::shared_ptr<DB::IDataType const> const&, std::__1::shared_ptr<DB::IDataType const> const&) const::'lambda1'(DB::Block&, std::__1::vector<unsigned long, std::__1::allocator<unsigned long> > const&, unsigned long), std::__1::allocator<DB::FunctionCast::prepare(std::__1::shared_ptr<DB::IDataType const> const&, std::__1::shared_ptr<DB::IDataType const> const&) const::'lambda1'(DB::Block&, std::__1::vector<unsigned long, std::__1::allocator<unsigned long> > const&, unsigned long)>, void (DB::Block&, std::__1::vector<unsigned long, std::__1::allocator<unsigned long> > const&, unsigned long)>::operator()(DB::Block&, std::__1::vector<unsigned long, std::__1::allocator<unsigned long> > const&, unsigned long&&) [gtests_dbms+4337478332]
                        /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.1.sdk/usr/include/c++/v1/functional:1558
     0x1073f9650        std::__1::__function::__func<DB::FunctionCast::prepare(std::__1::shared_ptr<DB::IDataType const> const&, std::__1::shared_ptr<DB::IDataType const> const&) const::'lambda1'(DB::Block&, std::__1::vector<unsigned long, std::__1::allocator<unsigned long> > const&, unsigned long), std::__1::allocator<DB::FunctionCast::prepare(std::__1::shared_ptr<DB::IDataType const> const&, std::__1::shared_ptr<DB::IDataType const> const&) const::'lambda1'(DB::Block&, std::__1::vector<unsigned long, std::__1::allocator<unsigned long> > const&, unsigned long)>, void (DB::Block&, std::__1::vector<unsigned long, std::__1::allocator<unsigned long> > const&, unsigned long)>::operator()(DB::Block&, std::__1::vector<unsigned long, std::__1::allocator<unsigned long> > const&, unsigned long&&) [gtests_dbms+4337473104]
                        /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.1.sdk/usr/include/c++/v1/functional:1732
     0x10739027c        std::__1::__function::__value_func<void (DB::Block&, std::__1::vector<unsigned long, std::__1::allocator<unsigned long> > const&, unsigned long)>::operator()(DB::Block&, std::__1::vector<unsigned long, std::__1::allocator<unsigned long> > const&, unsigned long&&) const [gtests_dbms+4337042044]
                        /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.1.sdk/usr/include/c++/v1/functional:1885
     0x10738f004        std::__1::function<void (DB::Block&, std::__1::vector<unsigned long, std::__1::allocator<unsigned long> > const&, unsigned long)>::operator()(DB::Block&, std::__1::vector<unsigned long, std::__1::allocator<unsigned long> > const&, unsigned long) const [gtests_dbms+4337037316]
                        /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.1.sdk/usr/include/c++/v1/functional:2560
     0x10738e908        DB::ExecutableFunctionCast::executeImpl(DB::Block&, std::__1::vector<unsigned long, std::__1::allocator<unsigned long> > const&, unsigned long) const [gtests_dbms+4337035528]
                        dbms/src/Functions/FunctionsConversion.h:2491
     0x10f63fb58        DB::IExecutableFunction::execute(DB::Block&, std::__1::vector<unsigned long, std::__1::allocator<unsigned long> > const&, unsigned long) const [gtests_dbms+4474075992]
                        dbms/src/Functions/IFunction.cpp:230
     0x105c9c8ac        DB::IFunctionBase::execute(DB::Block&, std::__1::vector<unsigned long, std::__1::allocator<unsigned long> > const&, unsigned long) const [gtests_dbms+4312975532]
                        dbms/src/Functions/IFunction.h:98
     0x10e5817e8        DB::ExpressionAction::execute(DB::Block&) const [gtests_dbms+4456519656]
                        dbms/src/Interpreters/ExpressionActions.cpp:336
     0x10e585784        DB::ExpressionActions::execute(DB::Block&) const [gtests_dbms+4456535940]
                        dbms/src/Interpreters/ExpressionActions.cpp:662
     0x10d319b54        DB::ExpressionBlockInputStream::readImpl() [gtests_dbms+4437220180]
                        dbms/src/DataStreams/ExpressionBlockInputStream.cpp:53
     0x10d3665a4        DB::IProfilingBlockInputStream::read(DB::PODArray<unsigned char, 4096ul, Allocator<false>, 15ul, 16ul>*&, bool) [gtests_dbms+4437534116]
                        dbms/src/DataStreams/IProfilingBlockInputStream.cpp:75
     0x10d366410        DB::IProfilingBlockInputStream::read() [gtests_dbms+4437533712]
                        dbms/src/DataStreams/IProfilingBlockInputStream.cpp:43
     0x10d319af4        DB::ExpressionBlockInputStream::readImpl() [gtests_dbms+4437220084]
                        dbms/src/DataStreams/ExpressionBlockInputStream.cpp:50
     0x10d3665a4        DB::IProfilingBlockInputStream::read(DB::PODArray<unsigned char, 4096ul, Allocator<false>, 15ul, 16ul>*&, bool) [gtests_dbms+4437534116]
                        dbms/src/DataStreams/IProfilingBlockInputStream.cpp:75
     0x10d366410        DB::IProfilingBlockInputStream::read() [gtests_dbms+4437533712]
                        dbms/src/DataStreams/IProfilingBlockInputStream.cpp:43
     0x106a35318        DB::tests::(anonymous namespace)::readBlock(std::__1::shared_ptr<DB::IBlockInputStream>) [gtests_dbms+4327232280]
                        dbms/src/TestUtils/ExecutorTestUtils.cpp:111
     0x106a351e4        DB::tests::ExecutorTest::executeStreams(std::__1::shared_ptr<tipb::DAGRequest> const&, std::__1::unordered_map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::vector<DB::ColumnWithTypeAndName, std::__1::allocator<DB::ColumnWithTypeAndName> >, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, std::__1::vector<DB::ColumnWithTypeAndName, std::__1::allocator<DB::ColumnWithTypeAndName> > > > >&, unsigned long) [gtests_dbms+4327231972]
                        dbms/src/TestUtils/ExecutorTestUtils.cpp:127
     0x106a354c0        DB::tests::ExecutorTest::executeStreams(std::__1::shared_ptr<tipb::DAGRequest> const&, unsigned long) [gtests_dbms+4327232704]
                        dbms/src/TestUtils/ExecutorTestUtils.cpp:132
     0x104e898a0        DB::tests::JoinExecutorTestRunner_agg_Test::TestBody() [gtests_dbms+4298217632]
                        dbms/src/Flash/tests/gtest_join_executor.cpp:370
     0x10c719db8        void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) [gtests_dbms+4424637880]
                        contrib/googletest/googletest/src/gtest.cc:2401
     0x10c6f27f8        void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) [gtests_dbms+4424476664]
                        contrib/googletest/googletest/src/gtest.cc:2437
     0x10c6f2754        testing::Test::Run() [gtests_dbms+4424476500]
                        contrib/googletest/googletest/src/gtest.cc:2473
     0x10c6f358c        testing::TestInfo::Run() [gtests_dbms+4424480140]
                        contrib/googletest/googletest/src/gtest.cc:2655
     0x10c6f4284        testing::TestCase::Run() [gtests_dbms+4424483460]
                        contrib/googletest/googletest/src/gtest.cc:2773
     0x10c6fad3c        testing::internal::UnitTestImpl::RunAllTests() [gtests_dbms+4424510780]
                        contrib/googletest/googletest/src/gtest.cc:4673
unknown file: Failure
C++ exception with description "DB::Exception" thrown in the test body.
[  FAILED  ] JoinExecutorTestRunner.agg (921 ms)
[----------] 1 test from JoinExecutorTestRunner (921 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (924 ms total)
[  PASSED  ] 0 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] JoinExecutorTestRunner.agg

 1 FAILED TEST
4. What is your TiFlash version? (Required)

master

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in dbms/src/Debug/astToExecutor.cpp at Aggregation::toTiPBExecutor, then review the JoinExecutorTestRunner.agg case in dbms/src/Flash/tests/gtest_join_executor.cpp. Run ./gtests_dbms --gtest_filter="JoinExecutorTestRunner.agg" and verify SUM output uses the intended unsigned field type while preserving NULL for SUM(NULL) without conversion errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.