Improve the integration of ctest and our buildbot

_From [timurrrr@google.com](https://code.google.com/u/timurrrr@google.com/) on April 28, 2011 06:20:12_

Current problems:

0) All the builds are actually red and we're not quite aware of this.
We should try hard to make "ctest" run green on all platforms. Otherwise buildbot usage
is suboptimal. As a temporary solution we should
- suppress all the unexpected false positives (and file bugs, of course)
- exclude flaky/hangy tests (nudge?) and fix them and re-enable
- relax the .out files to work on all Windows platforms
the same way (I'm talking about "4/5 leaks" Win version
differences on suppress tests),

1) {ctest = "build" + "run all tests"} is run as a single step.
as "ctest -VV -S /tests/runsuite.cmake,drmemory_only".
This doesn't fit well into the buildbot philosophy.
It would be better if we could separate "build" and "test" into separate commands,
like "make ALL_BUILD" and "make test" (or .sh/.bat scripts or whatever).

I believe this would also be very valuable for local testing.
Maybe this is already there but I don't know how to run it separately -
(I know "ctest run" but it didn't work quite well for me. Also, is there "ctest build"?)
Derek, can you please comment on this?

2) If possible, ctest should return non-zero code in case of any test failures.
If this is not possible - I can grep the logs and set non-zero code on the buildbot side
but it looks like a hack.

3) Individual test failures from "ctest" should be shown on the waterfall / build page.
Currently we only see an orange box (= no errors, only warnings)
with "stdio" link (leads to full ctest log - too much info and requires ctest knowledge)
and "warnings" link (leads to VS build warnings, see issue #367 )

For example, `grep "Test project\|\*\*\*Failed" \` gives much more
buildbot-friendly information (and user- too!), e.g.
Test project E:/b/build/slave/win_xp_drm_checkout/build/build_drmemory-dbg-32
5/20 Test `#5`: cs2bug ...........................**_Failed 7.61 sec
20/20 Test `#20`: nudge ............................**_Failed 18.16 sec
Test project E:/b/build/slave/win_xp_drm_checkout/build/build_drmemory-rel-32
5/20 Test `#5`: cs2bug ...........................**_Failed 2.67 sec
18/20 Test `#20`: nudge ............................**_Failed 1.48 sec
Test project E:/b/build/slave/win_xp_drm_checkout/build/build_drmemory-cyg-dbg-32
5/18 Test `#5`: cs2bug ...........................**_Failed 6.56 sec
15/18 Test `#12`: suppress .........................**_Failed 120.07 sec
16/18 Test `#13`: suppress-genoffs .................**_Failed 123.38 sec
17/18 Test `#14`: suppress-gensyms .................**_Failed 123.27 sec
18/18 Test `#18`: nudge ............................***Failed 204.63 sec
<\- this is easy to read. And I can easily extract the list of test failures for each test failure.

Derek, (how?) is it possible to get individual test logs from CTest so we easily access them
from the buildbot, without downloading logs as a .7z file (as we do now)

4) Why ctest at all?
Don't worry, I don't advocate to abandon it :) just out of curiosity and for my personal
"recipe of success for a new opensource project".

If (just imagine) you were starting from scratch - would you still use ctest?
Looks like many opensource projects tend to use buildbot, not ctest http://wiki.winehq.org/UnitTestSuites

_Original issue: http://code.google.com/p/drmemory/issues/detail?id=368_