Azure / Azure/azure-sdk-tools

[FEATURE REQ] Compare JSON Content By Key/Value, Not Text Diff

Open
#7,477 6 comments 0 reactions 0 assignees View on GitHub
Client
Dominant language
C#
Stars
135
Forks
260
Avg merge
1d 22h
Merged PRs (30d)
115

Description

### Library name

Azure.Core.TestFramework

### Please describe the feature.

## Description

Looks like in the playback mode, the test framework is comparing JSON content by text diff, which will cause false alerts when only the sequence of some properties has been changed.

### Example

We see repetitive alerts below after the SDK codes are refreshed.
```
Azure.Core.TestFramework.TestRecordingMismatchException : Unable to find a record for the request PUT https://management.azure.com/subscriptions/xxxxx/resourcegroups/teststorageRG-5248?api-version=2021-04-01
Header differences:
Body differences:
Request and response bodies do not match at index 2:
request: "{"tags":{"test":"env"}"
record: "{"location":"eastus2","
```

But actually the content is not changed semantically.

![MicrosoftTeams-image](https://user-images.githubusercontent.com/4516/149862591-0699ca73-7c02-4f4e-b7ca-4fe86f71e6f5.png)

### Solution

We should try to compare the JSON body content by key/value, not the text diff.

Contributor guide

Open the contributing guide

Research direction

Start by locating the Azure.Core.TestFramework playback logic that compares request and response bodies. Reproduce the PUT recording mismatch with JSON properties in a different order, then verify that semantically equivalent bodies no longer report a mismatch while genuinely different key/value content still does.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.