DynamoRIO / DynamoRIO/dynamorio

ASSERT: various asserts when running fleetbench

Open
#6,409 16 comments 0 reactions 1 assignee Claimed by @ivankyluk View on GitHub
Dominant language
C
Stars
3.2k
Forks
629
Avg merge
2d 15h
Merged PRs (30d)
31

Description

**Describe the bug**
We've been running the fleetbench tests internally and using the invariant checker to verify the results and have hit several asserts. This issue will cover the one we have managed to reproduce on the AArch64 CI machine with clang-12.

This assert manifests as:
```
Trace invariant failure in T1406984 at ref # 18217530 (1 instrs since timestamp 13343320636382108): Non-explicit control flow has no marker
Aborted (core dumped)
```

**To Reproduce**
```
wget https://github.com/DynamoRIO/dynamorio/releases/download/release_10.0.0/DynamoRIO-AArch64-Linux-10.0.0.tar.gz
tar -xzf DynamoRIO-AArch64-Linux-10.0.0.tar.gzwget https://github.com/bazelbuild/bazelisk/releases/download/v1.17.0/bazelisk-linux-arm64
chmod a+x bazelisk-linux-arm64git clone https://github.com/google/fleetbench.git
git -C fleetbench fetch origin v0.2.1
git -C fleetbench checkout v0.2.1
echo "build:opt --copt='-mcpu=cortex-A57'" >> fleetbench/.bazelrc
export DRUN=$(pwd)/DynamoRIO-AArch64-Linux-10.0.0/bin64/drrun
export OUTDIR=$(pwd)/output
mkdir ${OUTDIR}
cd fleetbench
GLIBC_TUNABLES=glibc.pthread.rseq=0 ../bazelisk-linux-arm64 run \
--run_under="${DRUN} -t drcachesim -offline -outdir ${OUTDIR} --" \
--config=clang --config=opt "fleetbench/swissmap:hot_swissmap_smoketest" && \
${DRUN} -t drcachesim -simulator_type invariant_checker \
-indir ${OUTDIR}/drmemtrace.hot_swissmap_smoketest.*
```

**Versions**
- These initial issues appeared in our head builds, but are reproducable with the latest release (10.0.0)
- These have been tested on ubuntu 20.04

**Additional context**

Additionally we are seeing the following asserts on our internal machines:

hot swissmap:
```
Trace invariant failure in T-1 at ref # 0 (0 instrs since timestamp 0): Serial schedule entry count does not match trace
```

cold swissmap:
```
Trace invariant failure in T1010 at ref # 89 (54 instrs since timestamp 13339766484591828): Timestamp does not increase monotonically
```

and when using sample windows:

fleetbench/tcmalloc:empirical_test:
```
Trace invariant failure in T7049 at ref # 3402633 (0 instrs since timestamp 13343308919845820): Syscall marker not placed after syscall instruction
```

fleetbench/swissmap:cold_swissmap_smoketest and fleetbench/swissmap:hot_swissmap_smoketest:
```
Trace invariant failure in T4279 at ref # 89 (54 instrs since timestamp 13343308657587587): Timestamp does not increase monotonically
```

fleetbench/tcmalloc:empirical_driver_test:
```
Trace invariant failure in TTrace invariant failure in T7919 at ref # Trace invariant failure in T7914 at ref # 9 (Trace invariant failure in T7954 at ref # 9 (0 instrs since timestamp 7945 at ref # 9 (0 instrs since timestamp 133433090606881269): 013343309095144172 instrs since timestamp 13343308960112710): An unfiltered thread should have at least 1 instruction (
): An unfiltered thread should have at least 1 instructionAn unfiltered thread should have at least 1 instruction0Trace invariant failure in T7929 at ref # 9 (0
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.