bug(datepicker): Date range picker incorrectly has matStartDateInvalid error after modification

#### Reproduction

Use StackBlitz to reproduce your issue: https://stackblitz.com/edit/angular-uotl3m-yzo7qt

Steps to reproduce:
1. Pick a date range in date picker.
2. Modify the start date using the input with a date after the end date
3. Modify the end date using the input with a date after the start date

#### Expected Behavior

The FormGroup is valid and not error message is show.

#### Actual Behavior

What behavior did you actually see?

The error message "Invalid start date" is show.

**If you change the version of material from "^11.0.0" to "11.0.0" there is no problemr**

#### Environment

- Angular: "^11.0.0
- CDK/Material: "^11.0.0
- Browser(s): Chrome
- Operating System (e.g. Windows, macOS, Ubuntu): windows 10

Start with the linked StackBlitz reproduction and follow the date range picker inputs through the start- and end-date modifications. Confirm the issue is fixed when changing the start date after the end date and then the end date leaves the FormGroup valid without a “matStartDateInvalid” error message.

Written by the indexing model from the issue text.