Azure / Azure/azure-sdk-tools

Test-Proxy Re-Record

Open
#13,976 0 comments 0 reactions 1 assignee Claimed by @joseharriaga View on GitHub
Asset-Sync Test-Proxy
Dominant language
C#
Stars
135
Forks
260
Avg merge
3d 1h
Merged PRs (30d)
144

Description

## Context

To update recordings for something even as simple as an API version update (with no functional recording changes), devs locally need to:

- Auth their local CLI to TME
- Remember how to deploy test resources, run the deployment
- Swap record mode to `record`, invoke recordings for the necessary tests to cover the new api version
- check recordings (though for stuff like this it is likely totally FUD and unnecessary to so)
- test-proxy push the results to assets repo, which updates assets.json
- push update to assets.json manually to the PR

This feature covers enabling these recordings updates _without _dev intervention.

## Proposal

The proposed https://github.com/openai/openai-dotnet start is a little bit awkward, because the recordings are stored locally. This means that the experience of updating the recordings will be a little bit more stilted in the openai-dotnet repo, but NOT insurmountable.

This is the effective flow, green being steps we want to add.

```mermaid
flowchart LR
A[Submit PR] --> B[Trigger LiveTest Runs]
B --> C[Collect Recordings]
C --> D[PR Recordings to original PR]

%% Colors
classDef blue fill:#1f6feb,stroke:#1f6feb,color:#ffffff;
classDef green fill:#2da44e,stroke:#2da44e,color:#ffffff;

class A,B blue;
class C,D green;
```

First we need to get the layout of these `livetest` pipelines:

- Save the static test run secrets as secrets within repo settings (for the selected static resources)
- Create a new github action that can be triggered against a PR
- The new github action should
- be trigger `pull_request_target` EXPLICITLY
- Check out the PR
- Run the specific tests in `Record mode`
- dump a `git diff` -> we expect minor changes for properly run record mode tests

1. ~~Speak with @hallipr and figure out if there is an existing github PAT available for `openai` org that is rotated regularly. This is the very first thing we need to nail down, as it will gate our PR efforts.~~ No pats exist right now.
2. How are they divided up in build yml?
3. Create a definition based upon the original livetest pipeline, this one, when triggered on a PR context, will also trigger additional steps into the `livetest` job that will detect and push a pr
4. Modify the input to the original livetest pipeline which _runs_ the tests to inject `record` mode from the special PR-triggered build def
5. After deploying resources, running tests, utilize the `- ${{ if }}` injected steps defined in 2) to actually invoke the `git diff`, and submit a PR using [a common piece of powershell you can crib off of](https://github.com/Azure/azure-sdk-for-python/blob/7611e0fbc6b727c4829fbb339230330418cebf22/eng/common/scripts/Submit-PullRequest.ps1#L4)
- [Example usage](https://github.com/Azure/azure-sdk-for-python/blob/7611e0fbc6b727c4829fbb339230330418cebf22/eng/common/pipelines/templates/archetype-typespec-emitter.yml#L494)
6. Expanding the project a bit. This could be the _default_ of all our livetest pipelines. The concern will merely be how we determine a material vs immaterial recording update. We can't simply say "ANY changes to the recordings" means stuff should get pushed, as often times timestamps and dates on requests will update (but be ignored by actual test code asserts). Therefore we will have to build a tool or copilot skill that can evaluate a diff and make a call whether to push it as a recording update or not.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.