google / google/fuzzbench

Git Checkout might not work if there are multiple copies of a repo.

Open
#1,908 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.2k
Forks
302
PR merge metrics
No merged PRs in 30d

Description

The `benchmark.yaml` file has a checkout hash that can be used to checkout a specific commit before running the benchmarks. We have observed that the checkout seems to fail spuriously in some benchmarks, like in `libjpeg-turbo_libjpeg_turbo_fuzzer`.

By [looking at the source code](https://github.com/google/fuzzbench/blob/7c70037a73d2cc66627c6109d53d20b3594f85c9/docker/benchmark-builder/checkout_commit.py#L92-L96), we found out that the code seems to go into each directory and try to checkout the specific commit, stopping at the first folder that succeeded.

This means that, in the case of libjpeg-turbo that has at least 3 folders (one for each branch), only one of them will be in the right commit. Furthermore, it seems like the order of the directory iteration is not stable, so it might change between runs.

We haven't looked to see which other benchmarks are affected.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.