Add Retries to Tests

needs grooming

- add retries to failed e2e tests
- accounts for flakiness in test environment
- avoid retry storms in mass-failure runs

Start by locating the end-to-end test runner and the failure-handling path for failed e2e tests. Determine how retries should account for test-environment flakiness without creating retry storms during mass failures, then verify the behavior with the relevant end-to-end test runs.

Written by the indexing model from the issue text.