DynamoRIO / DynamoRIO/drmemory

many false positive invalid heap args on 64-bit Chromium tests

Open
#1,839 1 comment 0 reactions 1 assignee Claimed by @derekbruening View on GitHub
Bug-FalsePositive Hotlist-Chrome Hotlist-Release OpSys-x64
Dominant language
C
Stars
2.7k
Forks
290
PR merge metrics
No merged PRs in 30d

Description

The 64-bit FYI test bot is now up and running, but there are many reported invalid heap args.

Xref #1831. These may similarly all be due to bad callstacks: xref #1222, #1833.

http://build.chromium.org/p/chromium.fyi/builders/Windows%20Tests%20%28DrMemory%20x64%29/builds/18

media:

```
[ RUN ] FFmpegDemuxerTest.Initialize_OpenFails
~~Dr.M~~
~~Dr.M~~ Error #1: INVALID HEAP ARGUMENT: allocated with Windows API layer, freed with C library layer
~~Dr.M~~ # 0 replace_free [e:\b\build\slave\win-builder\drmemory\common\alloc_replace.c:2706]
~~Dr.M~~ # 1 base.dll!base::MessagePumpDefault::~MessagePumpDefault [base\message_loop\message_pump_default.cc:23]
~~Dr.M~~ # 2 media.dll!media::FFmpegGlue::~FFmpegGlue [media\filters\ffmpeg_glue.cc:222]
~~Dr.M~~ # 3 media.dll!media::FFmpegDemuxer::~FFmpegDemuxer [media\filters\ffmpeg_demuxer.cc:763]
~~Dr.M~~ # 4 media::FFmpegDemuxer::`scalar deleting destructor'
~~Dr.M~~ # 5 media::FFmpegDemuxerTest::~FFmpegDemuxerTest [media\filters\ffmpeg_demuxer_unittest.cc:81]
~~Dr.M~~ # 6 media::FFmpegDemuxerTest_Initialize_OpenFails_Test::`scalar deleting destructor'
~~Dr.M~~ # 7 testing::TestInfo::Run [testing\gtest\src\gtest.cc:2662]
~~Dr.M~~ Note: @0:00:24.617 in thread 3732
[ OK ] FFmpegDemuxerTest.Initialize_OpenFails (72 ms)
```

ipc:

```
[ RUN ] IPCAttachmentBrokerPrivilegedWinTest.SendHandle
~~Dr.M~~
~~Dr.M~~ Error #1: INVALID HEAP ARGUMENT: allocated with Windows API layer, freed with C library layer
~~Dr.M~~ # 0 replace_free [e:\b\build\slave\win-builder\drmemory\common\alloc_replace.c:2706]
~~Dr.M~~ # 1 base.dll!base::File::TakePlatformFile [base\files\file_win.cc:31]
~~Dr.M~~ # 2 MSVCP120.dll!std::_Locinfo::_Locinfo_ctor +0x43 (0x000007fef205e788 )
~~Dr.M~~ # 3 MSVCP120.dll!std::_Locinfo::_Locinfo +0x9d (0x000007fef2028faa )
~~Dr.M~~ # 4 MSVCP120.dll!std::num_put<>::_Getcat +0x78 (0x000007fef2040b59 )
~~Dr.M~~ # 5 MSVCP120.dll!std::time_get<>::_Getvals<> +0x6f9 (0x000007fef2033a9e )
~~Dr.M~~ # 6 MSVCP120.dll!std::basic_ostream<>::operator<< +0x60 (0x000007fef203ac3d )
~~Dr.M~~ # 7 base.dll!logging::LogMessage::Init [base\logging.cc:753]
~~Dr.M~~ # 8 base.dll!logging::LogMessage::LogMessage [base\logging.cc:493]
~~Dr.M~~ # 9 `anonymous namespace'::CommonPrivilegedProcessMain [ipc\attachment_broker_privileged_win_unittest.cc:443]
~~Dr.M~~ #10 multi_process_function_list::InvokeChildProcessTest [testing\multiprocess_func_list.cc:51]
~~Dr.M~~ #11 base::TestSuite::Run [base\test\test_suite.cc:222]
~~Dr.M~~ #12 base::internal::Invoker<>::Run [base\bind_internal.h:350]
~~Dr.M~~ #13 base::`anonymous namespace'::LaunchUnitTestsInternal [base\test\launcher\unit_test_launcher.cc:206]
~~Dr.M~~ #14 base::LaunchUnitTests [base\test\launcher\unit_test_launcher.cc:450]
~~Dr.M~~ #15 main [ipc\run_all_unittests.cc:23]
~~Dr.M~~ Note: @0:00:01.000 in thread 196
[2760:196:1230/033853:2121519:INFO:attachment_broker_privileged_win_unittest.cc(443)] Privileged process start.
[2760:196:1230/033853:2121597:INFO:attachment_broker_privileged_win_unittest.cc(455)] Privileged process spinning run loop.
[2760:196:1230/033853:2121629:INFO:attachment_broker_privileged_win_unittest.cc(462)] Privileged process running callback.
[2760:196:1230/033853:2121629:INFO:attachment_broker_privileged_win_unittest.cc(464)] Privileged process finishing callback.
[2760:196:1230/033853:2121629:INFO:attachment_broker_privileged_win_unittest.cc(455)] Privileged process spinning run loop.
[2760:196:1230/033853:2121629:INFO:attachment_broker_privileged_win_unittest.cc(469)] Privileged process end.
[ OK ] IPCAttachmentBrokerPrivilegedWinTest.SendHandle (1649 ms)
```

There are many instances of invalid heap args involving RtlProcessFlsData:

ipc not visible in bot stdio, also base:

```
INVALID HEAP ARGUMENT: allocated with Windows API layer, freed with C library layer
#0 replace_free [e:\b\build\slave\win-builder\drmemory\common\alloc_replace.c:2706]
#1 base.dll!base::MessagePumpForIO::~MessagePumpForIO [base\message_loop\message_pump_win.cc:436]
#2 ntdll.dll!RtlProcessFlsData +0x83 (0x000000007735ac14 )
#3 ntdll.dll!LdrShutdownThread +0x4a (0x000000007737381b )
#4 KERNEL32.dll!BaseThreadInitThunk +0xc (0x00000000772259cd )
Note: @0:00:11.820 in thread 1956
```

Same thing in this particular test for net_:

```
[ RUN ] NetworkQualityEstimatorTest.TestKbpsRTTUpdates
~~888~~
~~888~~ Error #1: INVALID HEAP ARGUMENT: allocated with Windows API layer, freed with C library layer
~~888~~ # 0 replace_free [e:\b\build\slave\win-builder\drmemory\common\alloc_replace.c:2706]
~~888~~ # 1 base.dll!base::MessagePumpForIO::~MessagePumpForIO [base\message_loop\message_pump_win.cc:436]
~~888~~ # 2 ntdll.dll!RtlProcessFlsData +0x83 (0x000000007735ac14 )
~~888~~ # 3 ntdll.dll!LdrShutdownThread +0x4a (0x000000007737381b )
~~888~~ # 4 KERNEL32.dll!BaseThreadInitThunk +0xc (0x00000000772259cd )
~~888~~ Note: @0:00:25.870 in thread 888
[ OK ] NetworkQualityEstimatorTest.TestKbpsRTTUpdates (501 ms)
```

Plus a bunch of other tests:

```
[ RUN ] DiskCacheEntryTest.GetTimes
[ RUN ] DiskCacheEntryTest.SizeChanges
[ RUN ] DiskCacheEntryTest.NewEvictionDoomEntryNextToOpenEntry
[ RUN ] DiskCacheEntryTest.SimpleCacheInternalAsyncIO
[ RUN ] DiskCacheEntryTest.SimpleCacheDoomCreateRace
[ RUN ] URLFetcherTest.DontRetryOnNetworkChangedByDefault
[ RUN ] URLFetcherTest.FileTestOverwriteExisting
[ RUN ] EmbeddedTestServerThreadingTestInstantiation/EmbeddedTestServerThreadingTest.RunTest/4
```

Also in components, device, cloud_print, mojo_system, cast, extensions,
google_apis, and unit_.

Different bogus callstack in net_:

```
[ RUN ] ClientSocketPoolBaseTest.CancelBackupSocketAfterFinishingAllRequests
~~3608~~
~~3608~~ Error #7: INVALID HEAP ARGUMENT: allocated with Windows API layer, freed with C library layer
~~3608~~ # 0 replace_free [e:\b\build\slave\win-builder\drmemory\common\alloc_replace.c:2706]
~~3608~~ # 1 net.dll!net::HostResolverImpl::ProcTask::DoLookup [net\dns\host_resolver_impl.cc:729]
~~3608~~ # 2 ntdll.dll!RtlProcessFlsData +0x83 (0x000000007735ac14 )
~~3608~~ # 3 ntdll.dll!LdrShutdownThread +0x4a (0x000000007737381b )
~~3608~~ # 4 ntdll.dll!RtlExitUserThread +0x37 (0x00000000773736e8 )
~~3608~~ # 5 ntdll.dll!TpDisassociateCallback +0x67d (0x0000000077346a2e )
~~3608~~ # 6 KERNEL32.dll!BaseThreadInitThunk +0xc (0x00000000772259cd )
~~3608~~ Note: @0:03:44.033 in thread 3608
[ OK ] ClientSocketPoolBaseTest.CancelBackupSocketAfterFinishingAllRequests (388 ms)
```

Seems like MSVCR100D!_freefls should be on the callstack.
Xref #1278; xref #997 _getptd. Probably this is similar, with some
internal libc routine being called on alloc resulting in our first
interception being at the WinAPI layer.

Sure looks like a bogus callstack frame -- and maybe it becomes a
"potential error" w/ proper callstack?

In content_, missing some frames so doesn't match #960 suppression (xref #1831):

```
~~3124~~ Error #1: INVALID HEAP ARGUMENT: allocated with Windows API layer, freed with C library layer
~~3124~~ # 0 replace_free [e:\b\build\slave\win-builder\drmemory\common\alloc_replace.c:2706]
~~3124~~ # 1 KERNEL32.dll!BaseThreadInitThunk +0xc (0x00000000772259cd )
~~3124~~ Note: @0:00:14.966 in thread 3124
```

I can't repro any of these invalid heap args locally!

They may well only occur with a lack of libc symbols, so this is my template for attempting to reproduce:

```
% _NT_SYMBOL_PATH= bin64/drmemory -no_fetch_symbols -light -batch -pause_at_assert -dr d:/derek/dr/git/exports -- z:/derek/chromium/src/out/Release_x64/ipc_tests.exe --single-process-tests --gtest_filter=
```

I tried media and ipc individual and running full suite from bot, both w/ my build and w/ DrMem unpacked/.

Maybe it's best to just disable this feature on the bots? We already do this for Windows Release:

```
proc += ["-no_check_delete_mismatch"]
```

So we'd add -no_check_heap_mismatch.

Xref #1838: add new option -malloc_callstacks to obtain malloc info for mismatch errors in light mode, and turn on for chrome?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.