Add regex scrubbing

Open
#514 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
30/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Stale
Tech stack
python
Domain
data

Research direction

Locate the existing Stop Words/Keep Words and Consolidation options, then read how they process tokens and replacements. The work is done when both options support regex-based removal or retention and replacement such as the University example, with tests covering the resulting single tokens and the interaction with punctuation stripping.

Written by the indexing model from the issue text.

Description

enhancement Scrubbing

Nice enhancements would be to be able to to remove/keep strings based on regex pattern. My suggestion would be to add this functionality to the Stop Words/Keep Words option. A version should also be added to the Consolidation option. That way you could, for instance, change /University of (.+)\s/ to "University_of_$1". In other words, "University of Edinburgh" and "University of Kentucky" would become "University_of_Edinburgh" and "University_of_Kentucky", and each could be counted as a single token.

This kind of functionality shouldn't be too hard to add, though in the example above the user would have to be careful not to consolidate and then strip punctuation...

Dominant language
Python
Stars
123
Forks
20
No merged PRs in 30d

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from WheatonCS/Lexos

All issues in WheatonCS/Lexos

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.