chakra-core / chakra-core/ChakraCore

Test Suite Cleanup

Open
#6,445 1 comment 0 reactions 0 assignees View on GitHub
Dev Experience help wanted NiceToHave Task
Dominant language
JavaScript
Stars
9.3k
Forks
1.2k
PR merge metrics
No merged PRs in 30d

Description

There are various issues with the test suite that make it harder to maintain and use than it should be. Each of these should be addressed. However this obviously won't improve the experience for any users of ChakraCore it's a purely for developer/contributor experience hence this is low priority but still something to work on:

- [ ] Tests can't be run on release builds https://github.com/microsoft/ChakraCore/issues/6422
- [x] If a test file does nothing OR does not exist it is counted as a pass #6596
- [ ] To run tests: Linux & macOS use runtests.py, windows uses rl.exe - ideally should be consistent #6585 #6596
- [ ] Many tests have unnecessary baseline files when they could just throw exceptions for failures and print 'pass' for success - baselines are strictly necessary only when tracing actions that can't be synchronously monitored/checked within javascript
- [ ] There are 48 tests that don't use a baseline but print a multi-line output ('pass'/'passed' repeatedly)
- [ ] Success indicators vary "Pass", "Passed", "pass", "PASSED" etc. - should be consistent.
- [ ] Many tests are not well explained/don't clearly show what they're for
- [ ] Many test files could be combined into larger aggregate files - which would run faster and be easier to manage
- [ ] Baseline files all use CRLF line endings - ch outputs CRLF on windows but LF only on linux/macOS - currently runtests.py converts CRLF in the baselines to LF only upon every run - should update ch to do LF only and remove all CRs

#6444 makes some progress on some of these points though there is far more to be done.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.