corvus-dotnet / corvus-dotnet/Corvus.Testing

Improve per-feature function hosting

Open
#50 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
C#
Stars
13
Forks
11
PR merge metrics
No merged PRs in 30d

Description

The `FunctionsBindings` is designed for per-scenario function instantiation. Although this provides isolation between tests, it also tends to discourage large numbers of tests because it slows things down—we spin up a new function host for each scenario. (And since we currently have no support for dynamic port allocation when testing functions, there's no way to parallelize these tests despite the isolution that this model in principle offers.)

So in practice it's fairly common for tests to want to spin up a function at the SpecFlow feature level. There are two significant problems with this:

1. Tests have to roll their own support for this
2. We lose all standard output and standard error reporting

That second issue is mostly down to design problems with unit test frameworks in .NET: there's no good way to report problems that happen at a scope larger than within one single test. This is partly a problem caused by the design of the common test tooling, with the result that pretty much all test frameworks in .NET handle this poorly.

However, we could still do better than we do today. If the `FunctionsController` were to offer a way to access the "output so far", it would be possible for failing tests to report the entire output from the function. While this is likely to include a lot of irrelevant detail (because it will include output for _all_ tests run so far, not just the failing one) and will include a lot of repetition if multiple scenarios within a feature fail, it is significantly better than having no output at all, which is where we are today. (And in the happy path in which all tests fail, we wouldn't need to show the output, so most of the time it would make no difference.)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the FunctionsBindings and FunctionsController designs described in the issue, focusing on how per-scenario function hosts are created and how output is currently handled. Define a feature-level hosting path and an "output so far" access point; done means feature-scoped tests can use the host and failing tests can report accumulated function output.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.