agoda-com / agoda-com/dotnet-build-metrics

Capturing Test run data from Developer Local Machines

未关闭
#25 1 条评论 1 个 reaction 已指派 0 人 在 GitHub 查看
enhancement
主要语言
C#
星标
5
派生
4
PR 合并指标
30 天内没有已合并 PR

描述

# Feature request

## Type

- [x] - Enhancement - completely new feature
- [ ] - Improvement - make what we have better

## Is your feature request related to a problem?

We want to collect test run data from local machines (Laptops) to analyse and help improve the developer experience. We should be able to correlate this with data from CI to look for common behaviour that indicates poor local experience.

For example:
- People not running test on their local, pushing to CI for testing
- Specific test suites that only run on CI that people never run locally
- Test that take a long time to run locally when compare to CI, or both CI and local are long
- Tests that're repeatedly run on without code change to succeed, indicating flakiness (might be tricky)

## Describe the solution you'd like

Most of our projects use nunit, so we can start here.

Nunit has an Engine Extension that we can use to create an installable package.

https://docs.nunit.org/articles/nunit-engine/extensions/creating-extensions/Writing-Engine-Extensions.html

It requires these `.addins` files to register it (they just need to be a content item in the package tools folder see example [here](https://github.com/nunit/nunit-v2-result-writer/blob/5f56c2893d3c6bbab3285b8dd9bd4231b9f8fd7a/cake/packaging.cake#L62))

https://docs.nunit.org/articles/nunit-engine/extensions/Installing-Extensions.html

Then there is this extension Type we can use to listen to test events.

https://docs.nunit.org/articles/nunit-engine/extensions/creating-extensions/Event-Listeners.html

```csharp
public interface ITestEventListener
{
///
/// Handle a progress report or other event.
///
/// An XML progress report.
void OnTestEvent(string report);
}
```

The event takes a XML documents as string input so we'll need to parse the string and it'll be one of these event types

- Start of run - ``
- End of run - ``
- Start of a test suite - ``
- End of a test suite - ``
- Start of a test case - ``
- End of a test case - ``

We'll probably just need to capture the "End of a test case" and report back from there to the same ingester we are using for build metrics.

In short we should be able to create a contactless "install nuget package and go" experience for the developers, for fast onboarding.

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。