DynamoRIO / DynamoRIO/dynamorio

speed up Appveyor CI workflow

Open
#2,406 1 comment 0 reactions 0 assignees View on GitHub
Component-Tests Hotlist-ContinuousIntegration Maintainability OpSys-Windows
Dominant language
C
Stars
3.2k
Forks
629
Avg merge
2d 15h
Merged PRs (30d)
31

Description

Appveyor (#2145) has enabled us to move to a purely cloud-based workflow without each dev needing to locally test every platform. However, Appveyor's free solution for open-source projects has no build parallelization either within a job or between jobs, meaning there's a global serialized queue of all pull request and master commit jobs. Since each one takes ~26 minutes this can be a bottleneck when multiple devs are submitting PR's simultaneously. Contrast this to Travis which has both inter- and intra-build parallelism and which I got down to under 10 minutes by splitting the suites up.

VS2010 32-bit-only build with no tests is taking nearly 7 mins.
VS2013 32-bit + 64-bit (+ vps) build + tests is taking 19 mins.

Ideas for speeding it up:

### Go back to requiring each dev to run Windows tests locally

I think this is a non-starter.

### Drop VS2010 support

Abandon VS2010 support and only build VS2013 on Appveyor. VS2012 Update 1 and VS2013 can still target XP so moving to VS2013 could be done. Xref #1424. Moving to VS2013 would also shrink pdb files (xref #2167).

### Run random subset of tests for PR's that are not clearly windows-affecting

For PR's that don't change win32/ or suite/tests/*win32, run a random subset of tests instead of every test. Do this via ctest -I with a randomly chosen interval? Count on the merge to master running everything and the dev reverting on any Windows failure there.

But, shrinking the number of tests may not make much of a difference: the build time dominates.

Plus there's plenty of shared code not in win32/ that affects Windows.

### Augment or switch to another hosted CI on Windows?

Travis may eventually add Windows but it doesn't look very promising: https://github.com/travis-ci/travis-ci/issues/2104

There don't seem to be any other hosted CI that supported Windows.

### Set up our own VM's or real hardware for CI

We'd want to pick one of the CI agent software products that already integrates nicely with Github.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.