opensearch-project / opensearch-project/OpenSearch

[Feature Request] Log the actual exception for InternalTestCluster shardLock failure

Open
#13,628 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

:test enhancement flaky-test Other
Dominant language
Java
Stars
13.7k
Forks
3k
Avg merge
2d 23h
Merged PRs (30d)
108

Description

Is your feature request related to a problem? Please describe

When i troubleshoot a flaky test of https://github.com/opensearch-project/ml-commons/issues/2436

the error message report test failure as below, it tells that shard [.plugins-ml-config][0] is still locked for somehow, there is no reason/details logged. It's inconvenient to know why the shard is still locked.

java.lang.AssertionError: Shard [.plugins-ml-config][0] is still locked after 5 sec waiting
        at __randomizedtesting.SeedInfo.seed([CC4F8C51F54E62C2:CF7AF5C5193A2F89]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.opensearch.test.InternalTestCluster.assertAfterTest(InternalTestCluster.java:2773)

https://github.com/opensearch-project/OpenSearch/blob/cd55bca3ee32d5754018e994b1b87799c1bffc32/test/framework/src/main/java/org/opensearch/test/InternalTestCluster.java#L2769-L2774

Describe the solution you'd like

As shared lock will have a detail/reason of why shard been locked, it would be better we log the exception which contains the detail of existing lock and then fail the test case.

https://github.com/opensearch-project/OpenSearch/blob/cd55bca3ee32d5754018e994b1b87799c1bffc32/server/src/main/java/org/opensearch/env/NodeEnvironment.java#L728-L742

try {
    env.shardLock(id, "InternalTestCluster assert after test", TimeUnit.SECONDS.toMillis(5)).close();
} catch (ShardLockObtainFailedException ex) {
    // would like to log out the ex that will helps to troubleshoot
    // logger.error(ex);
    fail("Shard " + id + " is still locked after 5 sec waiting");
}

the log should be similar like below, lock already held for [starting shard] this is the reason why it still been locked.

org.opensearch.env.ShardLockObtainFailedException: [.plugins-ml-config][0]: obtaining shard lock for [InternalTestCluster assert after test] timed out after [5000ms], lock already held for [starting shard] with age [20365ms]
	at org.opensearch.env.NodeEnvironment$InternalShardLock.acquire(NodeEnvironment.java:877) ~[opensearch-2.15.0-SNAPSHOT.jar:2.15.0-SNAPSHOT]
Related component

Other

Describe alternatives you've considered

No response

Additional context

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in test/framework/src/main/java/org/opensearch/test/InternalTestCluster.java at assertAfterTest, then inspect the ShardLockObtainFailedException path in server/src/main/java/org/opensearch/env/NodeEnvironment.java. Verify the failure output includes the exception's lock details before the test fails; reproduce or run the affected InternalTestCluster test scenario to confirm the diagnostic message.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
testing-qa
Issue type
Feature
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.