Automated Testing

# My Cypress vs Playwright Benchmark

102 tests in Chrome

On Jenkins, single core:
Cypress: 4:30
Playwright: 1:58

M1 MacBook Pro (multi-core):
Cypress: 2:15
Playwright: 20 seconds!

## Best Practices

- [ ] [Jest and Cypress best practices](https://blog.sapegin.me/all/react-testing-1-best-practices/)
- [ ] [javascript-testing-best-practices](https://github.com/goldbergyoni/javascript-testing-best-practices)
- [ ] [ui-testing-best-practices](https://github.com/NoriSte/ui-testing-best-practices)
- [ ] [awesome-regression-testing](https://github.com/mojoaxel/awesome-regression-testing)
- [ ] Tag tests in their names such as `#E2E`, `#Integration`, etc. Then can filter the list as needed by "tag"
- [ ] Use at least 2 levels of `describe` nesting to group related tests and provide useful names in test output. `User -> add user -> happy path -> returns id` and `User -> add user -> non-admin -> throws error`
- [ ] Found a bug? Write test **first**.
- [ ] [Unit testing is overrated](https://tyrrrz.me/blog/unit-testing-is-overrated). [My summary](https://twitter.com/housecor/status/1281581927942717441)
- [ ] Fail test on console log or error: https://github.com/ValentinH/jest-fail-on-console or https://github.com/tkrotoff/throw-on or https://twitter.com/housecor/status/1622595548267319297

## Data

- [ ] Write atomic tests. Write the data you need for each test, then remove it. Avoid using the same data for multiple tests if the data can be changed.

## Principles

- [ ] [What a unit test shouldnt do](https://twitter.com/housecor/status/1604549844667875330)
- [ ] Write tests that mimic how your users interact with your code
- [ ] Mindset: "[Zero One Many Lots Oops](https://twitter.com/housecor/status/1264676613335257090)" - Helps consider edge cases.
- [ ] [Focus on use cases, not code coverage](https://kentcdodds.com/blog/how-to-know-what-to-test)
- [ ] 5 questions every test should answer:
- What is the unit under test (module, function, class, whatever)?
- What should it do? (Prose description)
- What was the actual output?
- What was the expected output?
- How do you reproduce the failure?

[Programmer tests should - Approach comparison](https://twitter.com/housecor/status/1146946168943009794)
- Minimize programmer waiting.
- Run reliably.
- Predict deployability.
- Respond to behavior changes.
- Not respond to structure changes.
- Be cheap to write.
- Be cheap to read.
- Be cheap to change.

[More general advice](https://software.rajivprab.com/2019/04/28/rethinking-software-testing-perspectives-from-the-world-of-hardware/)
- Test both black box (purely on specs) and white box (look at the implementation and let that inform the tests you write). White-box-testing can improve test coverage, by better testing edge cases. 
- Mindset: "This is so important, that we have to build an automated test suite for it. I don’t trust human testers to do the job."
- Automated tests minimize faulty assumptions. They're a safety net for refactors later when the programmer doesn't know about all the previous assumptions and verbally communicated requirements.
- [Store related things together](https://twitter.com/housecor/status/1187796430557196289). So keep tests alongside the system under test.

> "Write tests. Not too many. Mostly integration." - Guillermo Rauch

## Automated Testing

- [ ] Frameworks (Mocha, Jasmine, Tape, Jest, AVA)
- [ ] Jest
- [ ] `npm i @types/jest` to get autocomplete. [Thread](https://github.com/jest-community/vscode-jest/issues/440#issuecomment-483872233)
- [ ] [Expect Methods](https://jestjs.io/docs/en/expect#methods)
- [ ] [Mocking dependencies](https://thoughtbot.com/blog/mocking-react-components-with-jest) - Also, consider using [msw to mock in both app and tests](https://kentcdodds.com/blog/stop-mocking-fetch)
- [ ] Snapshot testing - Useful for regression protection. Assures output doesn't change for a given input. So useful for components, mock API responses, to alert if anything changes after upgrading dependencies, or code that you want to refactor that's not currently under test. But NOT useful for TDD. Also not as good as visual tests (Percy/Chromatic) for UI due to false negatives. ANY markup refactor breaks the snapshot.
- [ ] [Inline snapshots](https://jestjs.io/docs/en/snapshot-testing#inline-snapshots) (Prettier required)
- [ ] [Focused / Partial snapshots](https://twitter.com/kentcdodds/status/956372500413165570)
- [ ] [toThrowErrorMatchingSnapshot](https://jestjs.io/docs/en/expect#tothrowerrormatchingsnapshothint)
- [ ] [Snapshot diffs](https://www.robinwieruch.de/jest-snapshot-difference/)
- [ ] Or use Enzyme's find to snapshot a portion of render's output.
- [ ] Or consider [html-looks-like](https://github.com/staltz/html-looks-like)
- [ ] [Some critiques and alternatives](https://engineering.ezcater.com/the-case-against-react-snapshot-testing)
- [ ] Config and Enhancements
- [ ] [jest-extended for more assertions](https://github.com/jest-community/jest-extended)
- [ ] [Use .each for multiple values per test](https://itnext.io/reduce-unit-tests-boilerplate-with-jests-each-syntax-f5e48828437f) or try [jest-in-case](https://github.com/atlassian/jest-in-case)
- [ ] [Options for Visual Jest output](https://twitter.com/housecor/status/1533502353348902912)

## Cypress vs Playwright differences

- type -> fill
- cy -> page
- Add await
- get -> locator
- it -> test
- cy.viewport ->
- cy.scrollIntoView -> Playwright does automatically
- Testing lib queries have shorter names on Playwright, and getBy* doesn’t assert by default.
- (should.not.exist) -> tohavecount(0)
- In Playwright, search in section by chaining off the parent’s locator.
- Playwright it’s easy to forget to await.
- describe vs test.describe

## General Structure and Practices
- [ ] [The Test pyramid](https://martinfowler.com/bliki/TestPyramid.html) is a classic, but [there are many more types of testing](https://cdn-images-1.medium.com/max/1600/1*TRAT1glkPpDFhv-GHCFBNQ.jpeg).
- [ ] Arrange, Act, Assert
- [ ] [Wrap in describe blocks to arrange](https://medium.com/@me_37286/yoni-goldberg-javascript-nodejs-testing-best-practices-2b98924c9347)
- [ ] Test the public API. Avoid testing implementation details.
- [ ] Leverage the same static mock data used for your mock API. Or [for more variety, consider using Faker/Chance](https://gist.github.com/i0natan/105f4e42dcd1a7ae0cb2c3afc717bd31#file-realistic-js).
- [ ] When testing creates/update/deletes, make each test atomic. [Write the data you want to manipulate](https://gist.github.com/i0natan/218b321bbcb3a591ab2f26b8288463d3#file-independenttest-js).
- [ ] Consider [tagging tests](https://gist.github.com/i0natan/8a0f3ff14112745d66ae8a215ffde6e2#file-tag-test-js) so related tests can be run in isolation. (Use filter tests by name)
- [ ] [Create a centralized render function](https://gist.github.com/coryhouse/095a373a009606a84d58fd20af69d91f) that handles React Router, Redux providers, etc.
- [ ] [Custom render wrapper for each component with getters for each input ](https://gist.github.com/coryhouse/5103e68609ce251578ecc4adf20a0558)
- [ ] When you have a jest mock function, you should use toHaveBeenCalledTimes *and* toHaveBeenCalledWith. Even if it's called only once, you want to make sure that it's called no more or less than one time. And note that [order matters](https://twitter.com/kentcdodds/status/1162098140532498432)

## React libraries
- [ ] [react-dom/test-utils](https://reactjs.org/docs/test-utils.html) (primarily useful for act and simulate, though RTL wraps act and has fireEvent, which are alternatives)
- [ ] [act](https://github.com/threepointone/react-act-examples) and Simulate are the most useful pieces. For DOM queries, Enzyme and RTL below are better.
- [ ] [Enzyme](https://airbnb.io/enzyme/)
- [ ] [Many examples and tips for configuring Jest custom serializer to serialize Enzyme snapshot tests without needing to call toJson](https://hackernoon.com/testing-react-components-with-jest-and-enzyme-41d592c174f)
- [ ] Use MDN's [shallowUntilTarget](https://github.com/mozilla/addons-frontend/blob/58d1315409f1ad6dc9b979440794df44c1128455/tests/unit/helpers.js#L276) to shallow render down to a target component. Useful [when testing a component wrapped in HOC](https://hacks.mozilla.org/2018/04/testing-strategies-for-react-and-redux). [More info and related Enzyme feature request](https://github.com/airbnb/enzyme/issues/539#issuecomment-321638534) (With this shallow rendering approach, it is now possible to test UserProfile in isolation yet still dispatch Redux actions like a real application.)
- [ ] [React-testing-library](https://testing-library.com)
- [ ] [React hooks testing library](https://github.com/testing-library/react-hooks-testing-library) for testing your custom hooks. Or can create a component that uses your hook and then test your component.

## Testing Library Tips
- [ ] Prefer queries in [their suggested order](https://testing-library.com/docs/guide-which-query) Query the [accessibility object model](https://developer.mozilla.org/en-US/docs/Glossary/AOM)
- [ ] [How to test a table](https://twitter.com/housecor/status/1524778321728372738)
- [ ] [Learn (and target) accessible names](https://developer.paciellogroup.com/blog/2017/04/what-is-an-accessible-name/)
- [ ] [Common Mistakes](https://kentcdodds.com/blog/common-mistakes-with-react-testing-library)
- [ ] [Use Jest-dom](https://github.com/testing-library/jest-dom)
- [ ] Configure ESLint with `npm i -D eslint-plugin-testing-library eslint-plugin-jest-dom`
- [ ] [To query within an element/container, use `within`](https://testing-library.com/docs/dom-testing-library/api-helpers#within-and-getqueriesforelement-apis). [Example](https://testing-library.com/docs/vue-testing-library/cheatsheet#other) and [here](https://twitter.com/kentcdodds/status/1158791591047208960)

## Cypress Tips

- Check a11y https://github.com/mfrachet/cypress-audit or https://www.npmjs.com/package/cypress-axe
3 ways to handle login:

**More broadly, don't use the UI to build up state for each test. Instead, set it.**

For example, don't actually run real login for each step. Instead, simulate it:

- [ ] [Can write Cypress "unit" tests against Storybook stories too](https://www.cypress.io/blog/2021/05/19/cypress-x-storybook-2-0/)
- [ ] [Review Cypress Real World App](https://cypress.io/blog/2020/06/11/introducing-the-cypress-real-world-app)
- [ ] [Best practices from docs](https://docs.cypress.io/guides/references/best-practices.html)
- [ ] Reset **before** tests for reliability.
- [ ] localStorage and cookies are cleared before each test automatically. So don't reset in tests.
- [ ] [Mastering UI Testing slides](https://slides.com/noriste/voxxed-days-ticino-2019-mastering-ui-testing)
- [ ] [cypress-example-realworld](https://github.com/cypress-io/cypress-example-realworld) - Real world app with many Cypress tests
- [ ] [eslint-plugin-cypress](https://github.com/cypress-io/eslint-plugin-cypress) protects from common mistakes.
- [ ] [eslint-plugin-no-only-tests](https://www.npmjs.com/package/eslint-plugin-no-only-tests) - protects from accidentally committing tests with `it.only`.
- [ ] [Disable video (useful when running the CI-oriented `run` command locally)](https://github.com/cypress-io/cypress/issues/867)
- [ ] Use [Cypress Testing Library](https://testing-library.com/docs/cypress-testing-library/intro) so you can write accessible "testing library" style queries in Cypress.
- [ ] Sometimes you'll need to force a click if the element is styled in a way that Cypress thinks hides it: `cy.getByLabelText("Select role for gsparks1").click({force: true });`
- [ ] [Keep the dev tools open as the tests run so you can see errors and warnings from React](https://twitter.com/housecor/status/1188543966398091264) (such as calling setState on unmounted components or invalid propTypes)
- [ ] [Export strings from component](https://slides.com/noriste/voxxed-days-ticino-2019-mastering-ui-testing#/15/2/0) to avoid copy/pasting strings between tests and components.
- [ ] Use subject to query for an element inside a parent
- [ ] Create plain functions for repeated code in one file. Use custom commands for repeated code in multiple files. [Declare TS Types](https://docs.cypress.io/api/cypress-api/custom-commands.html#5-Write-TypeScript-definitions) for custom commands.
- [ ] [Add skip and only to the test runner](https://github.com/bahmutov/cypress-skip-and-only-ui)

```js
it.only("should find 630 users when filtered by role of Salesperson or None", () => {
cy.getByLabelText("Filter by Role")
.click()
.then(subject => {
cy.getByLabelText("Salesperson", { container: subject }).click();
cy.getByLabelText("None", { container: subject }).click();
});
cy.getByText("630 users");
});
```

**Instructions for downloading Cypress if corp network is blocking**:
1) Go to: https://download.cypress.io/desktop/4.2.0?platform=win32&arch=x64
to download the cypress package.
2) Copy the downloaded zip file to c:\temp
3) Run the command (In a non powershell command line window):

set DEBUG=cypress:cli && set CYPRESS_INSTALL_BINARY=C:/temp/cypress.zip && npm install cypress
in the VSCode terminal or a cmd window

4) Get the last cypress package run the command:
npm i @testing-library/cypress

### Cypress issues
Cypress isn't perfect.

- [Keyboard tab key isn't supported](https://github.com/cypress-io/cypress/issues/299), though [you can use a plugin](https://docs.cypress.io/api/commands/type#Tabbing) or [create a custom command](https://github.com/cypress-io/cypress/issues/299#issuecomment-380197761).
- [Can't switch in/out of iframes](https://github.com/cypress-io/cypress/issues/136).
- [Doesn't support hover](https://github.com/cypress-io/cypress/issues/10).
- [Lacks support for some other keyboard events](https://github.com/cypress-io/cypress/issues/311)
- [Strips content security policy headers](https://github.com/cypress-io/cypress/issues/1030)

## Testing Different File Types

- [ ] Stateless components
- [ ] Instance methods
- [ ] Extracting to pure functions
- [ ] [Extracting setState / functional setState](https://medium.freecodecamp.org/functional-setstate-is-the-future-of-react-374f30401b6b)
- [ ] Use the [object builder pattern](https://github.com/coryhouse/react-slingshot/blob/master/src/components/FuelSavingsForm.spec.js) to simplify test setup, and the [component template pattern](https://twitter.com/housecor/status/1047497249226465280) (useful for testing React components and Storybook)
- [ ] Redux
- [ ] Actions
- [ ] Reducers
- [ ] mapStateToProps (extract to pure func / selector)
- [ ] Connected components
- [ ] Store (integration test)

## In browser testing / Integration Testing

- [ ] [Cypress](https://docs.cypress.io/examples/examples/tutorials.html#Test-a-React-Todo-App)
- [Cypress vs Selenium](https://blog.logrocket.com/cypress-io-the-selenium-killer/)
- [Create custom "app actions" API to speed Cypress test initialization](https://www.cypress.io/blog/2019/01/03/stop-using-page-objects-and-start-using-app-actions/). [Shorter post here](https://applitools.com/blog/page-objects-app-actions-cypress/)
- Debug on CI via https://github.com/flotwig/cypress-log-to-output
- [ ] [Playwright](https://playwright.dev/) - A fork of Puppeteer created by Microsoft, to focus on automated testing. A [short comparison of Playwright vs Cypress](https://cathalmacdonnacha.com/cypress-vs-playwright-which-is-best-for-e2e-testing). (Note that Cypress now supports multiple domains and webkit, so the post is outdated. Here's another ([Playwright vs Cypress](https://www.browserstack.com/guide/playwright-vs-cypress) post. Note that [Playwright is faster than Cypress](https://blog.checklyhq.com/cypress-vs-selenium-vs-playwright-vs-puppeteer-speed-comparison/). If you are a beginner with testing and are looking for ease of installation and usage with solid docs, then Cypress is the way to go. However, if speed matters more (it's faster, it even runs tests in parallel locally, and you can run tests against multiple browsers at the same time), you need `hover` or `tab` support, you prefer Playwright's traditional JS approach (instead of Cypress' chaining), or you need Playwright's multi-tab support, then it's the right choice.
- [ ] [Convert Cypress to Playwright](https://github.com/playwright-community/cy2pw)
- [ ] [Selenium](https://www.seleniumhq.org/)
- [ ] Wrappers like [Nightwatch](http://nightwatchjs.org) or [Katalon](https://www.katalon.com/)
- [ ] [TestCafe](
https://devexpress.github.io/testcafe/)
- [ ] [Webdriver.io](https://webdriver.io/) - Used by Cerner. [Webdriver vs Cypress](https://www.browserstack.com/guide/cypress-vs-webdriverio)
- [ ] [Codecept.io](https://codecept.io/)
- [ ] Scenario Testing (use UI to tell a mock API what to return)

## Contract Testing (aka Consumer Driven Contract Testing)

Overview: https://www.linkedin.com/pulse/api-contract-testing-visual-guide-peter-thomas/

Test the client/server contract. Assure the API provides the expected responses, via tests. [Here's why it's useful](https://reflectoring.io/7-reasons-for-consumer-driven-contracts/).

**Tools**:
- [ ] [Pact](https://docs.pact.io/)
- [ ] [url-content-changes-checker](https://www.npmjs.com/package/url-content-changes-checker)
- [ ] Even simply Jest snapshots (to store the expected responses from each APIs)

**Benefits**:
1. Can reveal unused interfaces - throws an error when there's unused provider code. If no apps are consuming it, obviously the code can be deleted from the provider (the API).


## Visual Testing

- [ ] [Good examples of why visual testing is useful](https://hackernoon.com/youve-been-using-selenium-and-are-pretty-happy-with-it-right-8eeac1bc875c)
- [ ] [Many options via Storybook](https://storybook.js.org/testing/automated-visual-testing)
- [ ] Percy
- [ ] Chromatic
- [ ] Jest image snapshot https://github.com/americanexpress/jest-image-snapshot
- [ ] Happo
- [ ] [Applitools with Cypress](https://applitools.com/blog/cypress-cross-browser-testing)
- [ ] [Awesome regression testing list](https://github.com/mojoaxel/awesome-regression-testing)

## Performance Testing / Monitoring

- [ ] https://calibreapp.com/

## Property Based Testing

Generate valid random values and feed to tests. e.g. Pass random strings and ints to a func that accepts a string and int.
- [ ] [jsverify](https://github.com/jsverify/jsverify) , [Test Check](https://github.com/leebyron/testcheck-js), [Fast Check](https://github.com/dubzzz/fast-check#readme)

## Validation Testing
https://ealush.com/vest

## Mutation Testing

Change code and if it doesn't fail, that means you have holes in your test coverage.
- [ ] [Stryker](https://stryker-mutator.io/)

## Fuzz Testing / Randomized Testing / Test Generation (like Pex for C#, or [QuickCheck](https://en.wikipedia.org/wiki/QuickCheck) which popularized the idea)

- [ ] [TestCheck](https://github.com/leebyron/testcheck-js)
- [ ] [fast-check](https://github.com/dubzzz/fast-check#readme)
- [ ] [jsfuzz](https://github.com/fuzzitdev/jsfuzz)
- [ ] [jsverify](https://github.com/jsverify/jsverify)
- [ ] [Medium post](https://medium.com/@gabescholz/randomized-testing-in-javascript-without-lifting-a-finger-8d616d7048af)
- [ ] https://docs.microsoft.com/en-us/visualstudio/test/intellitest-manual/test-generation?view=vs-2019

## Recommended Reading
[Testing React components with Jest and Enzyme](https://hackernoon.com/testing-react-components-with-jest-and-enzyme-41d592c174f)
[Node and JS Testing Best Practices](https://medium.com/@me_37286/yoni-goldberg-javascript-nodejs-testing-best-practices-2b98924c9347)