DynamoRIO / DynamoRIO/dynamorio

tool.drcacheoff.analysis_unit_tests broken on ARM/AArch32

Open
#7,172 3 comments 0 reactions 0 assignees View on GitHub
OpSys-ARM
Dominant language
C
Stars
3.2k
Forks
629
Avg merge
2d 18h
Merged PRs (30d)
30

Description

In fact it sometimes passes, but very infrequently, about one time in 50 tries for me.

With 7203e72333c273b3ac132a26be537abd197d0888, when it passes, it finishes very quickly.

When it fails the output looks like this:
```
9: ----------------
9: Testing tool errors
...
9: [scheduler] Reading headers from inputs to find filetypes
9: [scheduler] Output 0 triggered a rebalance @0:
9: [analyzer] Creating 2 worker threads
9: [analyzer] Worker 0 starting on trace shard 0 stream is 0xd31748
9: [analyzer] Worker 1 starting on trace shard 1 stream is 0xd31920
9: [analyzer] Worker 0 hit shard memref error cpuid not supported on trace shard
9: [scheduler] Queue snapshot: inputs: 1 schedulable, 0 unscheduled, 4 eof
9: out #0 @28: running #-1; 0 in queue; 0 blocked
9: out #1 @50012: running #-1; 0 in queue; 0 blocked
9:
9: [scheduler] Queue snapshot: inputs: 1 schedulable, 0 unscheduled, 4 eof
9: out #0 @28: running #-1; 0 in queue; 0 blocked
9: out #1 @100012: running #-1; 0 in queue; 0 blocked
...
```

When it passes the corresponding part of the output looks like:
```
9: ----------------
9: Testing tool errors
...
9: [scheduler] Reading headers from inputs to find filetypes
9: [scheduler] Output 0 triggered a rebalance @0:
9: [analyzer] Creating 2 worker threads
9: [analyzer] Worker 0 starting on trace shard 0 stream is 0x1ae9748
9: [analyzer] Worker 1 starting on trace shard 1 stream is 0x1ae9920
9: [analyzer] Worker 0 hit shard memref error cpuid not supported on trace shard
9: [scheduler] Output 0 triggered a rebalance @28:
9: [scheduler] exiting early at input 4 with 0 live inputs left
9: [analyzer] Worker 1 finished trace shard
9: [scheduler] Unscheduled queue lock acquired : 2
9: [scheduler] Unscheduled queue lock contended : 0
9: [scheduler] Stats for output #0
```

The test seems to have been broken by f1b2d541de3d498be8a6104b2788ea7ecf1184ec (17 Sep 2024). Before then it seems to always pass. With that commit it nearly always fails with output that looks like this:
```
9: ----------------
9: Testing tool errors
9: [scheduler] 5 inputs
9: [scheduler] Reading headers from inputs to find filetypes
9: [scheduler] Output 0 triggered a rebalance @0:
9: [analyzer] Creating 2 worker threads
9: [analyzer] Worker 0 starting on trace shard 0 stream is 0x1f7f280
9: [analyzer] Worker 1 starting on trace shard 1 stream is 0x1f7f440
9: [analyzer] Worker 1 hit shard memref error cpuid not supported on trace shard
9: [scheduler] Output 0 triggered a rebalance @1115050074:
9: [scheduler] Output 0 triggered a rebalance @1130050074:
9: [scheduler] Output 0 triggered a rebalance @1145050074:
9: [scheduler] Output 0 triggered a rebalance @1160050074:
9: [scheduler] Output 0 triggered a rebalance @1175050075:
9: [scheduler] Output 0 triggered a rebalance @1190050075:
1/1 Test #9: tool.drcacheoff.analysis_unit_tests ...***Timeout 90.01 sec
```

On the rare occasions when it passes:
```
9: ----------------
9: Testing tool errors
9: [scheduler] 5 inputs
9: [scheduler] Reading headers from inputs to find filetypes
9: [scheduler] Output 0 triggered a rebalance @0:
9: [analyzer] Creating 2 worker threads
9: [analyzer] Worker 0 starting on trace shard 0 stream is 0x25c7280
9: [analyzer] Worker 1 starting on trace shard 1 stream is 0x25c7440
9: [analyzer] Worker 0 hit shard memref error cpuid not supported on trace shard
9: [scheduler] Output 0 triggered a rebalance @1323071107:
9: [analyzer] Worker 1 finished trace shard
9: [scheduler] Stats for output #0
...
```

@derekbruening , since this was your change and only a few months ago, do you have any ideas about this?

Contributor guide

Open the contributing guide

Research direction

Start by reproducing tool.drcacheoff.analysis_unit_tests on ARM/AArch32 and compare the output before and after commit f1b2d541de3d498be8a6104b2788ea7ecf1184ec. Trace the scheduler and analyzer behavior around rebalances, worker errors, and the timeout. Done means the test completes reliably rather than intermittently timing out.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
testing-qa, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.