byexamples / byexamples/byexample

Use the `regex` package in replacement of the built-in `re` module

Open
#128 4 comments 0 reactions 0 assignees View on GitHub
enhancement far in the future
Dominant language
Python
Stars
67
Forks
9
PR merge metrics
No merged PRs in 30d

Description

**Describe the feature you'd like**
The idea is to replace the standard [re](https://docs.python.org/3/library/re.html) module with the third party [regex](https://pypi.org/project/regex/) module.

There are a few reasons to do it:

- the was not intention of [releasing the GIL](https://bugs.python.org/issue23690) by `re` while `regex` can do it (and it can be enforced with `concurrent=True`). The lack of multithreading support by `re` prevented to use threads instead of multi-processes in `byexample` which are, obviously, more expensive.
- `byexample` uses heavily the regex engine but all the regex used in a single run are different, unique. This means that the traditional cache of `re` is pointless because it disappears after each run (`byexample` is restarted and as any process the OS frees its memory). That means that in each run `byexample` needs to re-compile every single regex which it is very expensive. Pickling is pointless because currently `re` pickles only the expression and it compiles it when it loads the pickle so we don't save any time. `regex`, however, supports pickling the bytecode directly. **Note:** we should test how much we win with this.
- despite of been heavily optimized, the regex created by `byexample` may lead to a catastrophic collapse (endless high CPU usage). `re` does not support *atomic groups* or *possessive qualifiers* that could [reduce the impact of a catastrophic backtracking](https://www.regular-expressions.info/catastrophic.html) (see #16). `re` neither supports timeouts. `regex` in the other hand supports all of them.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.