opensearch-project / opensearch-project/OpenSearch
[Feature Request] Log the actual exception for InternalTestCluster shardLock failure
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 13.7k
- Forks
- 3k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 108
Description
Is your feature request related to a problem? Please describe
When i troubleshoot a flaky test of https://github.com/opensearch-project/ml-commons/issues/2436
the error message report test failure as below, it tells that shard [.plugins-ml-config][0] is still locked for somehow, there is no reason/details logged. It's inconvenient to know why the shard is still locked.
java.lang.AssertionError: Shard [.plugins-ml-config][0] is still locked after 5 sec waiting
at __randomizedtesting.SeedInfo.seed([CC4F8C51F54E62C2:CF7AF5C5193A2F89]:0)
at org.junit.Assert.fail(Assert.java:89)
at org.opensearch.test.InternalTestCluster.assertAfterTest(InternalTestCluster.java:2773)
Describe the solution you'd like
As shared lock will have a detail/reason of why shard been locked, it would be better we log the exception which contains the detail of existing lock and then fail the test case.
try {
env.shardLock(id, "InternalTestCluster assert after test", TimeUnit.SECONDS.toMillis(5)).close();
} catch (ShardLockObtainFailedException ex) {
// would like to log out the ex that will helps to troubleshoot
// logger.error(ex);
fail("Shard " + id + " is still locked after 5 sec waiting");
}
the log should be similar like below, lock already held for [starting shard] this is the reason why it still been locked.
org.opensearch.env.ShardLockObtainFailedException: [.plugins-ml-config][0]: obtaining shard lock for [InternalTestCluster assert after test] timed out after [5000ms], lock already held for [starting shard] with age [20365ms]
at org.opensearch.env.NodeEnvironment$InternalShardLock.acquire(NodeEnvironment.java:877) ~[opensearch-2.15.0-SNAPSHOT.jar:2.15.0-SNAPSHOT]
Related component
Other
Describe alternatives you've considered
No response
Additional context
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in test/framework/src/main/java/org/opensearch/test/InternalTestCluster.java at assertAfterTest, then inspect the ShardLockObtainFailedException path in server/src/main/java/org/opensearch/env/NodeEnvironment.java. Verify the failure output includes the exception's lock details before the test fails; reproduce or run the affected InternalTestCluster test scenario to confirm the diagnostic message.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- testing-qa
- Issue type
- Feature
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 50/100