adoptium / adoptium/aqa-tests

Potential autoGen.mk "tee" hang if testcase file directs not closed or background processes do not complete

Open
#6,752 5 comments 1 reaction 0 assignees View on GitHub
Dominant language
HTML
Stars
155
Forks
346
Avg merge
2d 9h
Merged PRs (30d)
36

Description

**Symptom:**
Testcase appears to complete, and Teardown message appears, but "hang" occurs until Jenkins aborts stage with timeout:
```
17:32:21 TEST TEARDOWN:
17:32:21 Nothing to be done for teardown.
.... HANG ....
02:25:58 Cancelling nested steps due to timeout
02:25:58 Sending interrupt signal to process
02:26:11 Terminated
02:26:11 make[4]: *** [autoGen.mk:33: awt_custom_0] Error 143
02:26:11 make[4]: Leaving directory '/home/jenkins/workspace/Test_openjdk11_xxx_x86-64_linux/aqa-tests/xxx/interactives'
02:26:11 make[3]: *** [/home/jenkins/workspace/Test_openjdk11_xxx_x86-64_linux/aqa-tests/TKG/../TKG/settings.mk:362: awt_custom_0-interactives] Error 2
02:26:11 make[3]: Leaving directory '/home/jenkins/workspace/Test_openjdk11_xxx_x86-64_linux/aqa-tests/xxx'
02:26:11 make[2]: *** [/home/jenkins/workspace/Test_openjdk11_xxx_x86-64_linux/aqa-tests/TKG/../TKG/settings.mk:362: awt_custom_0-xxx] Error 2
02:26:11 make[2]: Leaving directory '/home/jenkins/workspace/Test_openjdk11_xxx_x86-64_linux/aqa-tests'
02:26:11 make[1]: *** [settings.mk:362: awt_custom_0-..] Error 2
02:26:11 make[1]: Leaving directory '/home/jenkins/workspace/Test_openjdk11_xxx_x86-64_linux/aqa-tests/TKG'
02:26:11 make: *** [makefile:62: _awt_custom_0] Error 2
02:26:11 script returned exit code 2
```
The use of "tee" in autoGen.mk can have consequences when testcase sub/background processes perhaps do not close all files or handles:
```
echo ""; echo "TEST TEARDOWN:"; \
$(TEST_TEARDOWN); \
} 2>&1 | tee -a $(Q)$(TESTOUTPUT)$(D)TestTargetResult$(Q);
```

Possible solution/workaround maybe use process redirect rather than pipe.
Essentially hang is instigated by a miss-behaving testcase leaving file handle or background process dangling. I believe in the above example it was possibly an Arctic sub-process or handle.

Contributor guide

Open the contributing guide

Research direction

Start in autoGen.mk, especially the TEST TEARDOWN command and its tee pipeline, then inspect how testcase and background processes close inherited file handles. Reproduce the Jenkins timeout with a testcase that leaves a process or handle open and compare the teardown behavior; done means the stage exits without hanging while preserving the test output.

Written by the indexing model from the issue text.

Assessment

Tech stack
shell
Domain
build-system, ci-cd, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.