New API RFC

Issues with the current API

• Multiple optional positional arguments in the checkA11y() method make it awkward to use and very difficult to extend. We have several open pull requests that are adding a new argument at the end.
• Requires too much setup to have violations summary either in the terminal or JSON that could be supplied to another too.
• Doesn't support Cypress chaining in the checkA11y() method, and uses a custom context argument. This makes the usage less idiomatic and doesn't allow integration with tools like Cypress Testing Library.
• No way to define defaults for the checkA11y() method, we have to pass things like violationCallback in every call or create a custom command.

Related issues: #40, #49, #62, #67, #68

New API proposal

1. Remove the context argument of the checkA11y() method, and rely on the Cypress chaining instead:

cy.checkA11y() // Check the whole document
cy.getByTestId('my-modal').checkA11y() // Check part of the document

2. Replace the rest of the arguments with an optional object of the following shape:

interface CypressAxeOptions {
  axeOptions?: axe.RunOptions, // This used to be `options` argument
  shouldFail?: (violations: RunResults[], results: RunResults[]) => boolean,  // Replaces `includedImpacts` and `skipFailures`
  reporters?: Reporter[]
}

type Reporter = (results: RunResults[]) => void

interface RunResults {
  filename: string, // Test file
  label?: string, // Label passed to checkA11y
  results: axe.Result[]
}

The defaults are going to be:

{
  axeOptions: undefined,
  shouldFail: (violations) => violations.length > 0,  // Fail on any number of violations
  reporters: [
   require('cypress-axe/cli-reporter')
  ]
}

Reporters are replacing the violationCallback: they are more flexible, have access to both passes and violations, and you could have global reporters that would run after all tests, or pass a reporter for a separate checkA11y() method call. The default reporter is printing an overall summary after all tests.

3. Add a new optional label to the checkA11y() method:

cy.checkA11y(options?: CypressAxeOptions, label?: string)

Similar to labels describe() or it() methods, to identify results of a particular call.

4. Add a new method to define defaults:

cy.configureCypressAxe(options: CypressAxeOptions)

It should accept the same object as the checkA11y() method, and set default values that could be overwritten in checkA11y().

I hope this covers all the use cases we have. I'm not sure that all the names are clear, so if you have any feedback, leave a comment ;-)

Start by reviewing the existing checkA11y API and the related issues #40, #49, #62, #67, and #68, then read the discussion for agreement on the proposed names and behavior. Done means the accepted API supports Cypress chaining, configurable defaults, labels, reporters, and the proposed options without the current positional-argument limitations.