Hunting for nondeterminism in our tools

From time to time, we notice nondeterminism in some of our tools. We try to fix it once it is identified. It would be nice to be able to automatically and periodically test for it. We should ideally test:
- Diffs in several runs of the same build.
- Diffs in runs of different builds (compilers, options, etc.).
- Diffs in runs on different systems (Linux/Windows).
Moreover, there might be some inherent nondeterminisms - time stamps, absolute paths, etc.

Right now, I have no idea how to do something like this. If anyone have any suggestion, please comment.

The issue names no files, tests, or entry points, so start by surveying the tool execution flow and the existing build and CI setup. Determine where repeated builds and cross-system runs could be compared, then define completion as periodic checks that report meaningful differences while accounting for timestamps, absolute paths, and other inherent nondeterminism.

Written by the indexing model from the issue text.