microsoft / microsoft/playwright
[Feature] advanced configurations for routeFromHAR
- Dominant language
- TypeScript
- Stars
- 96.3k
- Forks
- 6.5k
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 180
Description
# Background / Use cases
The `routeFromHAR` functionality encapsulates two behaviors. When `update` is `true`, it behaves as a custom network recorder, while when `update` is `false` (default) it serves as a network communication mock.
I would like to suggest several improvements:
# saving minimal data
today, each HAR entry contains a lot of unnecessary data - timing, HTTP version, request headers, and more.
while this data might be useful for analysis, most of it is not being used for network traffic replay.
Despite that omitting the unnecessary fields would make a deviation from the formal HAR specifications, I believe it is worth it in terms of clarity, bundle size, concise git differences, output predictability, and maintainability.
## example
before:
```js
{
"startedDateTime": "2023-03-05T05:09:17.557Z",
"time": 8.897,
"request": {
"method": "GET",
"url": "http://localhost:10102/api/pages?accountId=",
"httpVersion": "HTTP/1.1",
"cookies": [],
"headers": [
{ "name": "Accept", "value": "application/json, text/plain, */*" },
{ "name": "Accept-Language", "value": "en-US" },
{ "name": "Cookie", "value": "cookies....." },
{ "name": "Expect", "value": "202+location" },
{ "name": "Referer", "value": "http://localhost:port/path?accountId=id },
{ "name": "User-Agent", "value": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/110.0.5481.177 Safari/537.36" },
{ "name": "sec-ch-ua", "value": "" },
{ "name": "sec-ch-ua-mobile", "value": "?0" },
{ "name": "sec-ch-ua-platform", "value": "" },
{ "name": "x-client-request-id", "value": "8a887b6a-7638-4e0a-bf04-b4c97a78199e--4" }
],
"queryString": [
{
"name": "accountId",
"value": ""
}
],
"headersSize": -1,
"bodySize": -1
},
"response": {
"status": 200,
"statusText": "OK",
"httpVersion": "HTTP/1.1",
"cookies": [],
"headers": [
{ "name": "X-Powered-By", "value": "Express" },
{ "name": "cache-control", "value": "no-store" },
{ "name": "connection", "value": "close" },
{ "name": "content-encoding", "value": "gzip" },
{ "name": "content-type", "value": "application/json; charset=utf-8" },
{ "name": "date", "value": "Sun, 05 Mar 2023 05:09:17 GMT" },
{ "name": "strict-transport-security", "value": "max-age=31536000" },
{ "name": "transfer-encoding", "value": "chunked" },
{ "name": "vary", "value": "Accept-Encoding" },
{ "name": "x-content-type-options", "value": "nosniff" },
{ "name": "x-permitted-cross-domain-policies", "value": "None" },
{ "name": "x-xss-protection", "value": "1; mode=block" }
],
"content": {
"size": -1,
"mimeType": "application/json; charset=utf-8",
"text": "{\"pages\":[{\"id\":\"1234\",\"name\":\"a new created test project\",\"createdAt\":\"2023-03-05T05:09:18.0612396+00:00\",\"updatedAt\":\"2023-03-05T05:09:18.0612396+00:00\",\"source\":\"filesystem\",\"status\":\"New\",\"metadata\":{}},{\"id\":\"00000251727294129115\",\"name\":\"with implementation\",\"createdAt\":\"2023-01-29T15:37:50.8846963+00:00\",\"updatedAt\":\"2023-01-29T15:38:07.5206989+00:00\",\"source\":\"filesystem\",\"status\":\"None\",\"metadata\":{}},{\"id\":\"00000251727302447488\",\"name\":\"no implementation\",\"createdAt\":\"2023-01-29T13:19:12.5111176+00:00\",\"updatedAt\":\"2023-01-29T15:37:23.8938755+00:00\",\"source\":\"filesystem\",\"status\":\"None\",\"metadata\":{}}]}"
},
"headersSize": -1,
"bodySize": -1,
"redirectURL": ""
},
"cache": {},
"timings": { "send": -1, "wait": -1, "receive": 8.897 }
}
```
after:
```js
{
"request": {
"method": "GET",
"url": "http://localhost:10102/api/pages?accountId="
]
},
"response": {
"status": 200,
"statusText": "OK",
"httpVersion": "HTTP/1.1",
"cookies": [],
"headers": [
{ "name": "X-Powered-By", "value": "Express" },
{ "name": "cache-control", "value": "no-store" },
{ "name": "connection", "value": "close" },
{ "name": "content-encoding", "value": "gzip" },
{ "name": "content-type", "value": "application/json; charset=utf-8" },
{ "name": "date", "value": "Sun, 05 Mar 2023 05:09:17 GMT" },
{ "name": "strict-transport-security", "value": "max-age=31536000" },
{ "name": "transfer-encoding", "value": "chunked" },
{ "name": "vary", "value": "Accept-Encoding" },
{ "name": "x-content-type-options", "value": "nosniff" },
{ "name": "x-permitted-cross-domain-policies", "value": "None" },
{ "name": "x-xss-protection", "value": "1; mode=block" }
],
"content": {
"mimeType": "application/json; charset=utf-8",
"text": "{\"pages\":[{\"id\":\"1234\",\"name\":\"a new created test project\",\"createdAt\":\"2023-03-05T05:09:18.0612396+00:00\",\"updatedAt\":\"2023-03-05T05:09:18.0612396+00:00\",\"source\":\"filesystem\",\"status\":\"New\",\"metadata\":{}},{\"id\":\"00000251727294129115\",\"name\":\"with implementation\",\"createdAt\":\"2023-01-29T15:37:50.8846963+00:00\",\"updatedAt\":\"2023-01-29T15:38:07.5206989+00:00\",\"source\":\"filesystem\",\"status\":\"None\",\"metadata\":{}},{\"id\":\"00000251727302447488\",\"name\":\"no implementation\",\"createdAt\":\"2023-01-29T13:19:12.5111176+00:00\",\"updatedAt\":\"2023-01-29T15:37:23.8938755+00:00\",\"source\":\"filesystem\",\"status\":\"None\",\"metadata\":{}}]}"
}
}
}
```
# excludeUrl
today, the `page.routeFromHAR` method receives `URL` option that defines which URLs should be included in the resulting HAR file.
I suggest we should have `excludeUrl` property. the property would allow users to exclude specific path(s) from recording.
## example
before:
```js
page.routeFromHAR(
`__tests__/networks_cache/${step}.har`,
{
url: /api|css|png|tff/, // user tries to list all URLs that could be memorized in the HAR file
}
);
```
after
```js
page.routeFromHAR(
`__tests__/networks_cache/${step}.har`,
{
url: /.*/, // user can use a wildcard
excludeUrl: "http://localhost/bundle.js" // user exclude specific path
}
);
```
# smart matching algorithm
[Playwright docs](https://playwright.dev/docs/network#replaying-from-har) state that:
> HAR replay matches URL and HTTP method strictly. For POST requests, it also matches POST payloads strictly. If multiple recordings match a request, the one with the most matching headers is picked
Would it be possible to grant users more control over he match algorithm?
## example
```js
page.routeFromHAR(
`__tests__/networks_cache/${step}.har`,
{
url: /api/,
urlMatching: {
"GET": {
matchLocation: "ignoreOrigin",
matchHeaders: false,
},
"POST": {
matchPayload: false
},
"delete": {
match: (harEntry, currRequest) => {
return userDefinedMatchScoringFunction(harEntry, currRequest);
}
}
}
}
);
```
# summery
Thank you for considering my suggestions!
Please note that my first suggestion is a breaking change - it would omit data from the HAR file, and there is a chance that some users rely on the timing/headers data in there.
I would love to hear your opinion and get some feedback before implementing any changes or submitting PRs.
Thank you for maintaining Playwright! I love this tool.
Noam
Contributor guide
Research direction
Start with the routeFromHAR API and the network replay documentation linked in the issue. Compare the proposed minimal HAR output, excludeUrl option, and matching controls against current behavior, then define scoped acceptance tests for recording, exclusion, and matching.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- api, networking, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100