microsoft / microsoft/playwright

[Feature] advanced configurations for routeFromHAR

Open
#21,405 18 comments 40 reactions 0 assignees View on GitHub
P3-collecting-feedback
Dominant language
TypeScript
Stars
96.3k
Forks
6.5k
Avg merge
1d 6h
Merged PRs (30d)
180

Description

# Background / Use cases
The `routeFromHAR` functionality encapsulates two behaviors. When `update` is `true`, it behaves as a custom network recorder, while when `update` is `false` (default) it serves as a network communication mock.
I would like to suggest several improvements:

# saving minimal data
today, each HAR entry contains a lot of unnecessary data - timing, HTTP version, request headers, and more.
while this data might be useful for analysis, most of it is not being used for network traffic replay.
Despite that omitting the unnecessary fields would make a deviation from the formal HAR specifications, I believe it is worth it in terms of clarity, bundle size, concise git differences, output predictability, and maintainability.
## example
before:
```js
{
"startedDateTime": "2023-03-05T05:09:17.557Z",
"time": 8.897,
"request": {
"method": "GET",
"url": "http://localhost:10102/api/pages?accountId=",
"httpVersion": "HTTP/1.1",
"cookies": [],
"headers": [
{ "name": "Accept", "value": "application/json, text/plain, */*" },
{ "name": "Accept-Language", "value": "en-US" },
{ "name": "Cookie", "value": "cookies....." },
{ "name": "Expect", "value": "202+location" },
{ "name": "Referer", "value": "http://localhost:port/path?accountId=id },
{ "name": "User-Agent", "value": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/110.0.5481.177 Safari/537.36" },
{ "name": "sec-ch-ua", "value": "" },
{ "name": "sec-ch-ua-mobile", "value": "?0" },
{ "name": "sec-ch-ua-platform", "value": "" },
{ "name": "x-client-request-id", "value": "8a887b6a-7638-4e0a-bf04-b4c97a78199e--4" }
],
"queryString": [
{
"name": "accountId",
"value": ""
}
],
"headersSize": -1,
"bodySize": -1
},
"response": {
"status": 200,
"statusText": "OK",
"httpVersion": "HTTP/1.1",
"cookies": [],
"headers": [
{ "name": "X-Powered-By", "value": "Express" },
{ "name": "cache-control", "value": "no-store" },
{ "name": "connection", "value": "close" },
{ "name": "content-encoding", "value": "gzip" },
{ "name": "content-type", "value": "application/json; charset=utf-8" },
{ "name": "date", "value": "Sun, 05 Mar 2023 05:09:17 GMT" },
{ "name": "strict-transport-security", "value": "max-age=31536000" },
{ "name": "transfer-encoding", "value": "chunked" },
{ "name": "vary", "value": "Accept-Encoding" },
{ "name": "x-content-type-options", "value": "nosniff" },
{ "name": "x-permitted-cross-domain-policies", "value": "None" },
{ "name": "x-xss-protection", "value": "1; mode=block" }
],
"content": {
"size": -1,
"mimeType": "application/json; charset=utf-8",
"text": "{\"pages\":[{\"id\":\"1234\",\"name\":\"a new created test project\",\"createdAt\":\"2023-03-05T05:09:18.0612396+00:00\",\"updatedAt\":\"2023-03-05T05:09:18.0612396+00:00\",\"source\":\"filesystem\",\"status\":\"New\",\"metadata\":{}},{\"id\":\"00000251727294129115\",\"name\":\"with implementation\",\"createdAt\":\"2023-01-29T15:37:50.8846963+00:00\",\"updatedAt\":\"2023-01-29T15:38:07.5206989+00:00\",\"source\":\"filesystem\",\"status\":\"None\",\"metadata\":{}},{\"id\":\"00000251727302447488\",\"name\":\"no implementation\",\"createdAt\":\"2023-01-29T13:19:12.5111176+00:00\",\"updatedAt\":\"2023-01-29T15:37:23.8938755+00:00\",\"source\":\"filesystem\",\"status\":\"None\",\"metadata\":{}}]}"
},
"headersSize": -1,
"bodySize": -1,
"redirectURL": ""
},
"cache": {},
"timings": { "send": -1, "wait": -1, "receive": 8.897 }
}
```
after:
```js
{
"request": {
"method": "GET",
"url": "http://localhost:10102/api/pages?accountId="
]
},
"response": {
"status": 200,
"statusText": "OK",
"httpVersion": "HTTP/1.1",
"cookies": [],
"headers": [
{ "name": "X-Powered-By", "value": "Express" },
{ "name": "cache-control", "value": "no-store" },
{ "name": "connection", "value": "close" },
{ "name": "content-encoding", "value": "gzip" },
{ "name": "content-type", "value": "application/json; charset=utf-8" },
{ "name": "date", "value": "Sun, 05 Mar 2023 05:09:17 GMT" },
{ "name": "strict-transport-security", "value": "max-age=31536000" },
{ "name": "transfer-encoding", "value": "chunked" },
{ "name": "vary", "value": "Accept-Encoding" },
{ "name": "x-content-type-options", "value": "nosniff" },
{ "name": "x-permitted-cross-domain-policies", "value": "None" },
{ "name": "x-xss-protection", "value": "1; mode=block" }
],
"content": {
"mimeType": "application/json; charset=utf-8",
"text": "{\"pages\":[{\"id\":\"1234\",\"name\":\"a new created test project\",\"createdAt\":\"2023-03-05T05:09:18.0612396+00:00\",\"updatedAt\":\"2023-03-05T05:09:18.0612396+00:00\",\"source\":\"filesystem\",\"status\":\"New\",\"metadata\":{}},{\"id\":\"00000251727294129115\",\"name\":\"with implementation\",\"createdAt\":\"2023-01-29T15:37:50.8846963+00:00\",\"updatedAt\":\"2023-01-29T15:38:07.5206989+00:00\",\"source\":\"filesystem\",\"status\":\"None\",\"metadata\":{}},{\"id\":\"00000251727302447488\",\"name\":\"no implementation\",\"createdAt\":\"2023-01-29T13:19:12.5111176+00:00\",\"updatedAt\":\"2023-01-29T15:37:23.8938755+00:00\",\"source\":\"filesystem\",\"status\":\"None\",\"metadata\":{}}]}"
}
}
}
```

# excludeUrl
today, the `page.routeFromHAR` method receives `URL` option that defines which URLs should be included in the resulting HAR file.
I suggest we should have `excludeUrl` property. the property would allow users to exclude specific path(s) from recording.
## example
before:
```js
page.routeFromHAR(
`__tests__/networks_cache/${step}.har`,
{
url: /api|css|png|tff/, // user tries to list all URLs that could be memorized in the HAR file
}
);
```
after
```js
page.routeFromHAR(
`__tests__/networks_cache/${step}.har`,
{
url: /.*/, // user can use a wildcard
excludeUrl: "http://localhost/bundle.js" // user exclude specific path
}
);
```
# smart matching algorithm
[Playwright docs](https://playwright.dev/docs/network#replaying-from-har) state that:

> HAR replay matches URL and HTTP method strictly. For POST requests, it also matches POST payloads strictly. If multiple recordings match a request, the one with the most matching headers is picked

Would it be possible to grant users more control over he match algorithm?
## example
```js
page.routeFromHAR(
`__tests__/networks_cache/${step}.har`,
{
url: /api/,
urlMatching: {
"GET": {
matchLocation: "ignoreOrigin",
matchHeaders: false,

},
"POST": {
matchPayload: false
},
"delete": {
match: (harEntry, currRequest) => {
return userDefinedMatchScoringFunction(harEntry, currRequest);
}
}
}
}
);
```
# summery
Thank you for considering my suggestions!
Please note that my first suggestion is a breaking change - it would omit data from the HAR file, and there is a chance that some users rely on the timing/headers data in there.
I would love to hear your opinion and get some feedback before implementing any changes or submitting PRs.

Thank you for maintaining Playwright! I love this tool.
Noam

Contributor guide

Open the contributing guide

Research direction

Start with the routeFromHAR API and the network replay documentation linked in the issue. Compare the proposed minimal HAR output, excludeUrl option, and matching controls against current behavior, then define scoped acceptance tests for recording, exclusion, and matching.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
api, networking, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.