[BUG] The Gradle test-retry plugin is flaky for tests with permutations (DataProvider)

Willingness to contribute

No. I cannot contribute a bug fix at this time.

Venice version

All

System information

All

Describe the problem

When using a DataProvider to inject multiple permutations of params into a test, and that test happens to be flaky, the retries do not always work. The reason seems to be that in addition to the test function name and params being part of the name, there is some "permutation index" number that is also inserted, but the value of that number is not deterministic. Oftentimes, it starts at zero, and then increments by two for each subsequent permutations, but sometimes it follows a different sequence. The Gradle test-retry plugin attempts to match tests of different attempts by their full name, which includes this non-deterministic number, and that can make the match fail, in which case the test-retry plugin gives up. The test name, including this number, may be generated by TestNG, though that is not clear. Whether the fix should belong within TestNG, or test-retry, or elsewhere is unknown for now.

Tracking information

No response

Code to reproduce bug

No response

What component(s) does this bug affect?

• Controller: This is the control-plane for Venice. Used to create/update/query stores and their metadata.
• Router: This is the stateless query-routing layer for serving read requests.
• Server: This is the component that persists all the store data.
• VenicePushJob: This is the component that pushes derived data from Hadoop to Venice backend.
• VenicePulsarSink: This is a Sink connector for Apache Pulsar that pushes data from Pulsar into Venice.
• Thin Client: This is a stateless client users use to query Venice Router for reading store data.
• Fast Client: This is a stateful client users use to query Venice Server for reading store data.
• Da Vinci Client: This is an embedded, stateful client that materializes store data locally.
• Alpini: This is the framework that fast-client and routers use to route requests to the storage nodes that have the data.
• Samza: This is the library users use to make nearline updates to store data.
• Admin Tool: This is the stand-alone client used for ad-hoc operations on Venice.
• Scripts: These are the various ops scripts in the repo.

Start by investigating the Gradle test-retry plugin with TestNG DataProvider tests and the generated names for parameter permutations. There is no reproduction code, file, or test named in the issue, so first establish a minimal reproducible case and determine whether the behavior belongs to TestNG or test-retry. Done means flaky permutations are matched reliably across retry attempts.