Enhance VSTestSettings with all vstest.console.exe command line options
Nobody has claimed this yet.
- Dominant language
- C#
- Stars
- 4.2k
- Forks
- 778
- Avg merge
- 3h 37m
- Merged PRs (30d)
- 21
Description
What You Are Seeing?
Cake VSTestSettings supports only a subset of vstest.console.exe console options for version 15.5 -->
https://cakebuild.net/api/Cake.Common.Tools.VSTest/VSTestSettings vs "vstest.console.exe console -?"
What is Expected?
Cake VSTestSettings should support ALL vstest.console.exe console options (e.g. Port, Blame, Framework as string or enhanced enum)
or
Provide a generic API to inject other properties like in the "MSBuildSettings" "WithProperty"
What version of Cake are you using?
0.23.0
Are you running on a 32 or 64 bit system?
64 bit
What environment are you running on? Windows? Linux? Mac?
Windows
Are you running on a CI Server? If so, which one?
Local and TFS Build Agent
How Did You Get This To Happen? (Steps to Reproduce)
Compare
https://cakebuild.net/api/Cake.Common.Tools.VSTest/VSTestSettings
vs
"vstest.console.exe console -?"
Output Log
vstest.console.exe -?
Microsoft (R) Test Execution Command Line Tool Version 15.5.0
Copyright (c) Microsoft Corporation. All rights reserved.
Usage: vstest.console.exe [Arguments] [Options] [[--] ...]]
Description: Runs tests from the specified files.
Arguments:
[TestFileNames]
Run tests from the specified files. Separate multiple test file names
by spaces.
Examples: mytestproject.dll
mytestproject.dll myothertestproject.exe
Options:
--Tests|/Tests:
Run tests with names that match the provided values. To provide multiple
values, separate them by commas.
Examples: /Tests:TestMethod1
/Tests:TestMethod1,testMethod2
--TestCaseFilter|/TestCaseFilter:
Run tests that match the given expression.
is of the format Operator[|&]
where Operator is one of =, != or ~ (Operator ~ has 'contains'
semantics and is applicable for string properties like DisplayName).
Parenthesis () can be used to group sub-expressions.
Examples: /TestCaseFilter:"Priority=1"
/TestCaseFilter:"(FullyQualifiedName~Nightly
|Name=MyTestMethod)"
--Framework|/Framework:
Target .Net Framework version to be used for test execution.
Valid values are ".NETFramework,Version=v4.5.1", ".NETCoreApp,Version=v1.0" etc.
Other supported values are Framework35, Framework40, Framework45 and FrameworkCore10.
--Platform|/Platform:
Target platform architecture to be used for test execution.
Valid values are x86, x64 and ARM.
--Settings|/Settings:
Settings to use when running tests.
RunSettings arguments:
Arguments to pass runsettings configurations through commandline. Arguments may be specified as name-value pair of the form [name]=[value] after "-- ". Note the spa
ce after --.
Use a space to separate multiple [name]=[value].
More info on RunSettings arguments support: https://aka.ms/vstest-runsettings-arguments
-lt|--ListTests|/lt|/ListTests:
Lists discovered tests from the given test container.
--Parallel|/Parallel
Specifies that the tests be executed in parallel. By default up
to all available cores on the machine may be used.
The number of cores to use may be configured using a settings file.
--TestAdapterPath|/TestAdapterPath
This makes vstest.console.exe process use custom test adapters
from a given path (if any) in the test run.
Example /TestAdapterPath:
--Blame|/Blame
Runs the test in blame mode. This option is helpful in isolating the problematic test causing test host crash. It creates an output file in the current directory as
"Sequence.xml", that captures the order of execution of test before the crash.
--Diag|/Diag:
Enable verbose logs for test platform.
Logs are written to the provided file.
--logger|/logger:<Logger Uri/FriendlyName>
Specify a logger for test results. For example, to log results into a
Visual Studio Test Results File (TRX) use /logger:trx [;LogFileName=]
Creates file in TestResults directory with given LogFileName.
Change the verbosity level for console logger. Allowed values for verbosity: quiet, minimal and normal.
Example: /logger:console;verbosity=<Defaults to "minimal">
To publish test results to Team Foundation Server, use TfsPublisher as shown below
Example: /logger:TfsPublisher;
Collection=<team project collection url>;
BuildName=<build name>;
TeamProject=<team project name>
[;Platform=<Defaults to "Any CPU">]
[;Flavor=<Defaults to "Debug">]
[;RunTitle=<title>]
--ResultsDirectory|/ResultsDirectory
Test results directory will be created in specified path if not exists.
Example /ResultsDirectory:
--ParentProcessId|/ParentProcessId:
Process Id of the Parent Process responsible for launching current process.
--Port|/Port:
The Port for socket connection and receiving the event messages.
-?|--Help|/?|/Help
Display this usage message.
--Collect|/Collect:
Enables data collector for the test run. More info here : https://aka.ms/vstest-collect
--InIsolation|/InIsolation
Runs the tests in an isolated process. This makes vstest.console.exe
process less likely to be stopped on an error in the tests, but tests
may run slower.
@
Read response file for more options.
To run tests:
>vstest.console.exe tests.dll
To run tests with additional settings such as data collectors:
>vstest.console.exe tests.dll /Settings:Local.RunSettings
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the VSTestSettings API and compare its current properties with the vstest.console.exe options listed in the issue. Review how similar settings are exposed in the Cake API, then determine whether the supported options or a generic property mechanism is the intended scope. Done means the required options can be passed through and are covered by the relevant tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- testing, tooling
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100