Python random tests are failing but CI is passing
- Dominant language
- C
- Stars
- 1.5k
- Forks
- 212
- PR merge metrics
- No merged PRs in 30d
Description
The `regex_test.py` and `regex_test_neg.py` scripts are failing but they don't return a non-zero exit code. This causes the Makefile to continue on like nothing went wrong thinking that everything passed. This can be fixed by adding the following lines to the end of the regex python scripts:
```python
if nfails != 0:
sys.exit(1)
```
The problem is that this will cause the CI to fail because the tests currently aren't passing, see alll the "FAIL" tests in this run: https://github.com/kokke/tiny-regex-c/runs/1939275804?check_suite_focus=true
```
[..snip..]
pattern '[1-5-]+[-1-2]-[-]': FAIL : doesn't match '-5--34522125-3131---' as expected [0x2d, 0x35, 0x2d, 0x2d, 0x33, 0x34, 0x35, 0x32, 0x32, 0x31, 0x32, 0x35, 0x2d, 0x33, 0x31, 0x33, 0x31, 0x2d, 0x2d, 0x2d].
FAIL : doesn't match '31433--3-33325121--' as expected [0x33, 0x31, 0x34, 0x33, 0x33, 0x2d, 0x2d, 0x33, 0x2d, 0x33, 0x33, 0x33, 0x32, 0x35, 0x31, 0x32, 0x31, 0x2d, 0x2d].
FAIL : doesn't match '12-32342324113--12--' as expected [0x31, 0x32, 0x2d, 0x33, 0x32, 0x33, 0x34, 0x32, 0x33, 0x32, 0x34, 0x31, 0x31, 0x33, 0x2d, 0x2d, 0x31, 0x32, 0x2d, 0x2d].
FAIL : doesn't match '455354554121-4515---' as expected [0x34, 0x35, 0x35, 0x33, 0x35, 0x34, 0x35, 0x35, 0x34, 0x31, 0x32, 0x31, 0x2d, 0x34, 0x35, 0x31, 0x35, 0x2d, 0x2d, 0x2d].
FAIL : doesn't match '---5551121353354--2--' as expected [0x2d, 0x2d, 0x2d, 0x35, 0x35, 0x35, 0x31, 0x31, 0x32, 0x31, 0x33, 0x35, 0x33, 0x33, 0x35, 0x34, 0x2d, 0x2d, 0x32, 0x2d, 0x2d].
FAIL : doesn't match '215535-35551-442--' as expected [0x32, 0x31, 0x35, 0x35, 0x33, 0x35, 0x2d, 0x33, 0x35, 0x35, 0x35, 0x31, 0x2d, 0x34, 0x34, 0x32, 0x2d, 0x2d].
FAIL : doesn't match '5-342342452523-52-41--' as expected [0x35, 0x2d, 0x33, 0x34, 0x32, 0x33, 0x34, 0x32, 0x34, 0x35, 0x32, 0x35, 0x32, 0x33, 0x2d, 0x35, 0x32, 0x2d, 0x34, 0x31, 0x2d, 0x2d].
FAIL : doesn't match '55515455412-1--' as expected [0x35, 0x35, 0x35, 0x31, 0x35, 0x34, 0x35, 0x35, 0x34, 0x31, 0x32, 0x2d, 0x31, 0x2d, 0x2d].
FAIL : doesn't match '2-3---' as expected [0x32, 0x2d, 0x33, 0x2d, 0x2d, 0x2d].
FAIL : doesn't match '444-2-3412--45433-1--' as expected [0x34, 0x34, 0x34, 0x2d, 0x32, 0x2d, 0x33, 0x34, 0x31, 0x32, 0x2d, 0x2d, 0x34, 0x35, 0x34, 0x33, 0x33, 0x2d, 0x31, 0x2d, 0x2d].
[..snip..]
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with regex_test.py and regex_test_neg.py, then inspect how the Makefile invokes them. Run the scripts to observe the reported FAIL cases and verify their exit status. Done means non-passing tests return a non-zero status so the Makefile and CI no longer report success when failures occur.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ci-cd, testing-qa
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100