kokke / kokke/tiny-regex-c

Python random tests are failing but CI is passing

Open
#66 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
1.5k
Forks
212
PR merge metrics
No merged PRs in 30d

Description

The `regex_test.py` and `regex_test_neg.py` scripts are failing but they don't return a non-zero exit code. This causes the Makefile to continue on like nothing went wrong thinking that everything passed. This can be fixed by adding the following lines to the end of the regex python scripts:

```python
if nfails != 0:
sys.exit(1)
```

The problem is that this will cause the CI to fail because the tests currently aren't passing, see alll the "FAIL" tests in this run: https://github.com/kokke/tiny-regex-c/runs/1939275804?check_suite_focus=true

```
[..snip..]
pattern '[1-5-]+[-1-2]-[-]': FAIL : doesn't match '-5--34522125-3131---' as expected [0x2d, 0x35, 0x2d, 0x2d, 0x33, 0x34, 0x35, 0x32, 0x32, 0x31, 0x32, 0x35, 0x2d, 0x33, 0x31, 0x33, 0x31, 0x2d, 0x2d, 0x2d].
FAIL : doesn't match '31433--3-33325121--' as expected [0x33, 0x31, 0x34, 0x33, 0x33, 0x2d, 0x2d, 0x33, 0x2d, 0x33, 0x33, 0x33, 0x32, 0x35, 0x31, 0x32, 0x31, 0x2d, 0x2d].
FAIL : doesn't match '12-32342324113--12--' as expected [0x31, 0x32, 0x2d, 0x33, 0x32, 0x33, 0x34, 0x32, 0x33, 0x32, 0x34, 0x31, 0x31, 0x33, 0x2d, 0x2d, 0x31, 0x32, 0x2d, 0x2d].
FAIL : doesn't match '455354554121-4515---' as expected [0x34, 0x35, 0x35, 0x33, 0x35, 0x34, 0x35, 0x35, 0x34, 0x31, 0x32, 0x31, 0x2d, 0x34, 0x35, 0x31, 0x35, 0x2d, 0x2d, 0x2d].
FAIL : doesn't match '---5551121353354--2--' as expected [0x2d, 0x2d, 0x2d, 0x35, 0x35, 0x35, 0x31, 0x31, 0x32, 0x31, 0x33, 0x35, 0x33, 0x33, 0x35, 0x34, 0x2d, 0x2d, 0x32, 0x2d, 0x2d].
FAIL : doesn't match '215535-35551-442--' as expected [0x32, 0x31, 0x35, 0x35, 0x33, 0x35, 0x2d, 0x33, 0x35, 0x35, 0x35, 0x31, 0x2d, 0x34, 0x34, 0x32, 0x2d, 0x2d].
FAIL : doesn't match '5-342342452523-52-41--' as expected [0x35, 0x2d, 0x33, 0x34, 0x32, 0x33, 0x34, 0x32, 0x34, 0x35, 0x32, 0x35, 0x32, 0x33, 0x2d, 0x35, 0x32, 0x2d, 0x34, 0x31, 0x2d, 0x2d].
FAIL : doesn't match '55515455412-1--' as expected [0x35, 0x35, 0x35, 0x31, 0x35, 0x34, 0x35, 0x35, 0x34, 0x31, 0x32, 0x2d, 0x31, 0x2d, 0x2d].
FAIL : doesn't match '2-3---' as expected [0x32, 0x2d, 0x33, 0x2d, 0x2d, 0x2d].
FAIL : doesn't match '444-2-3412--45433-1--' as expected [0x34, 0x34, 0x34, 0x2d, 0x32, 0x2d, 0x33, 0x34, 0x31, 0x32, 0x2d, 0x2d, 0x34, 0x35, 0x34, 0x33, 0x33, 0x2d, 0x31, 0x2d, 0x2d].
[..snip..]
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with regex_test.py and regex_test_neg.py, then inspect how the Makefile invokes them. Run the scripts to observe the reported FAIL cases and verify their exit status. Done means non-passing tests return a non-zero status so the Makefile and CI no longer report success when failures occur.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ci-cd, testing-qa
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.