microsoft / microsoft/MIEngine

Test infrastructure: Add a custom editor for DAR tests

Open
#1,332 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C#
Stars
859
Forks
233
Avg merge
1d 6h
Merged PRs (30d)
3

Description

Problem

It can be painful to create tests in DAR. This proposal seeks to improve two problems with authoring/maintaining DAR tests:

  1. Make it easier to turn the actual responses from a debug adapter into C# code for a DAR-based test
  2. When a DAR test fails with a log entry like the following, make it easier to understand why the 'actual' and 'expected' don't match.

Example log entry:

DebugAdapterRunner.DARException : Test command execution failed...
Failure = Expected response not found before timeout.
Expected = 
1. Not Found: {"success":true,"body":{"stackFrames":[{"id":1000,"name":"[External Code]"},{"id":1001,"name":"^This\\ will\\ fail\\ to\\ match$"},{"id":1002,"name":"Debuggee\\.Main\\(string\\[]\\ args\\)"}]}}

Actual Responses =
1. {"success":true,"message":null,"request_seq":5,"command":"stackTrace","body":{"stackFrames":[{"id":1000,"name":"[External Code]","line":0,"column":0,"presentationHint":"subtle"},{"id":1001,"name":"[Exception] Exceptions4Debuggee.exe!AsyncTest.C(string messageFormat) Line 51","source":{"name":"debuggee.cs","path":"C:\\dd\\Concord\\src\\vsdbg\\VsDbg-UITests\\Debuggees\\Exceptions4Debuggee\\debuggee.cs","sourceReference":0,"checksums":[{"algorithm":"SHA256","checksum":"b2ef5210e5727249981edef3380f6f4d34a1cb776683ec762cc21096f6d81a4a"}]},"line":51,"column":9,"endLine":51,"endColumn":87,"instructionPointerReference":"0x00007FFE0AA619D4","moduleId":1001},{"id":1002,"name":"[Exception] Exceptions4Debuggee.exe!AsyncTest.B(string messageFormat) Line 43","source":{"name":"debuggee.cs","path":"C:\\dd\\Concord\\src\\vsdbg\\VsDbg-UITests\\Debuggees\\Exceptions4Debuggee\\debuggee.cs","sourceReference":0,"checksums":[{"algorithm":"SHA256","checksum":"b2ef5210e5727249981edef3380f6f4d34a1cb776683ec762cc21096f6d81a4a"}]},"line":43,"column":9,"endLine":43,"endColumn":32,"instructionPointerReference":"0x00007FFE0AA615CE","moduleId":1001},{"id":1003,"name":"[Exception] Exceptions4Debuggee.exe!AsyncTest.A(string messageFormat) Line 36","source":{"name":"debuggee.cs","path":"C:\\dd\\Concord\\src\\vsdbg\\VsDbg-UITests\\Debuggees\\Exceptions4Debuggee\\debuggee.cs","sourceReference":0,"checksums":[{"algorithm":"SHA256","checksum":"b2ef5210e5727249981edef3380f6f4d34a1cb776683ec762cc21096f6d81a4a"}]},"line":36,"column":9,"endLine":36,"endColumn":32,"instructionPointerReference":"0x00007FFE0AA6122E","moduleId":1001},{"id":1004,"name":"[External Code]","line":0,"column":0,"presentationHint":"subtle"},{"id":1005,"name":"Exceptions4Debuggee.exe!AsyncTest.Run(System.Func<string> messageFormatFactory) Line 25","source":{"name":"debuggee.cs","path":"C:\\dd\\Concord\\src\\vsdbg\\VsDbg-UITests\\Debuggees\\Exceptions4Debuggee\\debuggee.cs","sourceReference":0,"checksums":[{"algorithm":"SHA256","checksum":"b2ef5210e5727249981edef3380f6f4d34a1cb776683ec762cc21096f6d81a4a"}]},"line":25,"column":13,"endLine":25,"endColumn":45,"instructionPointerReference":"0x00007FFE0AA60DF4","moduleId":1001},{"id":1006,"name":"[External Code]","line":0,"column":0,"presentationHint":"subtle"}],"totalFrames":7},"running":false,"refs":null,"seq":24,"type":"response"}


Command = {"args":{"threadId":26140,"startFrame":0,"levels":20},"Name":"stackTrace","ExpectedResponses":[{"Response":{"success":true,"body":{"stackFrames":[{"id":1000,"name":"^\\[This\\ will\\ fail\\ to\\ match]$","presentationHint":"subtle"},{"id":1001,"name":"^\\[Exception]\\ Exceptions4Debuggee\\.exe!AsyncTest\\.C\\(string\\ messageFormat\\)\\ Line\\ 51$","source":{"name":"^debuggee\\.cs$","path":".+\\\\Exceptions4Debuggee\\\\debuggee\\.cs$"},"line":51,"column":9,"endLine":51,"endColumn":87},{"id":1002,"name":"^\\[Exception]\\ Exceptions4Debuggee\\.exe!AsyncTest\\.B\\(string\\ messageFormat\\)\\ Line\\ 43$","source":{"name":"^debuggee\\.cs$","path":".+\\\\Exceptions4Debuggee\\\\debuggee\\.cs$"},"line":43},{"id":1003,"name":"^\\[Exception]\\ Exceptions4Debuggee\\.exe!AsyncTest\\.A\\(string\\ messageFormat\\)\\ Line\\ 36$","source":{"name":"^debuggee\\.cs$","path":".+\\\\Exceptions4Debuggee\\\\debuggee\\.cs$"},"line":36},{"id":1004,"name":"^\\[External\\ Code]$","presentationHint":"subtle"},{"id":1005,"name":"^Exceptions4Debuggee\\.exe!AsyncTest\\.Run\\(System\\.Func<string>\\ messageFormatFactory\\)\\ Line\\ 25$","source":{"name":"^debuggee\\.cs$","path":".+\\\\Exceptions4Debuggee\\\\debuggee\\.cs$"},"line":25},{"id":1006,"name":"^\\[External\\ Code]$","presentationHint":"subtle"}]}},"Match":null,"IgnoreOrder":false}]}

Proposal

We should have a WPF application that makes it easier to author and debug DAR based tests. A rough sketch is below:

| Paste From Exception Text | Copy As C# | Update Expected from Actual |

Status: <NO MATCH FOUND | Match Found | No expected text | syntax error>

Expected:
+-----------------------------------------------------------+
|                                                           |
|  <-- Editable rich text block with expected response -->       |
|                                                           |
+-----------------------------------------------------------+

[X] Ignore order

Actual:
+-----------------------------------------------------------+
|                                                           |
|  <-- Editable rich text block with actual response   -->       |
|                                                           |
+-----------------------------------------------------------+

Where:

  • 'Paste From Exception Text', 'Copy As C#', 'Update Expected From Actual' are buttons
  • 'Status' is a non-editable textbox
  • 'Ignore order' is a checkbox

The primary function of the app would be to match up the 'Expected' and 'Actual' text and report if they matched. It should do so
by refactoring or borrowing code from DebugAdapterRunner.DebugAdapterCommand.Run.

This program should also expose three other bits of functionality:

1: The 'Paste From Exception Text' button would look at the clipboard for text of the form
Expected = ...\nActual Responses =\n1 ...[\n2. ...] and would replace the current "Expected"
and "Actual" tetxt block with that content. This should include reformatting the JSON to add indentations so that it is readable.

2: Text highlighting for regular expressions. Every string is DAR is treated as a regular expression. Every string in the 'Expected' block should be highlighted in green, if at least one match was found, or red, if no match was found. If the cursor is in one of these regular expressions, that has a match, then it should get a yellow highlight in the 'Actual' block.

3: The 'Copy As C#' would take the content of the 'Expected' text block and reformat it as C# using anonymous types. For example, if the current Expected text is:

{
    "success": true,
    "body": {
        "variables": [
            {
                "name": "param1\\ \\[int\\]",
                "value": "^1$"
            }
        ]
    }
}

This should produce:

new 
{
    success = true,
    body = new
    {
        variables = new dynamic[]
        {
            new
            {
                name = @"param1\ \[int\]",
                value = @"^1$"
            }
        }
    }
}

4: Implement 'Update Expected From Actual' -- take the json from the 'actual' box, and paste it into the 'excepted' box, turning any string into a regex that would match just that value using:

        static string EscapeFullString(string value)
        {
            return "^" + Regex.Escape(value) + "$";
        }

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading DebugAdapterRunner.DebugAdapterCommand.Run in test/DebugAdapterRunner/DebugAdapterCommand.cs and reviewing the DAR test format and failure output described in the issue. Define the WPF editor around parsing exception text, matching expected and actual responses, and generating C#; it is done when the proposed actions, highlighting, and update behavior work for DAR tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
desktop, devtools, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.