Add character filtering to the preprocess script.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39
- Forks
- 7
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 5
Description
During our testing with character filtering there was one system that consistently improved the results (even if only slightly). It would be good to include that capability into the preprocess script. 'Alphabet' in this issue is used to refer to the those characters which are acceptable in the source or target data. For some experiments we may have a shared alphabet between the source and data, for other experiments the alphabets will be different.
This function is expected to be used only in the preprocessing of parent experiments. In those experiments with a large corpus there can be thousands of characters in the data, most of them are just noise.
The function would:
-
Read a character_filtering.yml file that specifies how the data is to be filtered.
-
Provide an interactive mode that would create the character_filtering.yml file.
-
Scan the source and target data as specified in the config.yml file and show the frequencies of every character in each.
-
Allow the user to specify the number of characters to use in the source and target alphabets, and write the resulting alphabet to the character_filtering.yml file.
-
The user would be able to make modifications to the character_filtering.yml file in order to specify changes to the alphabets.
For example certain rare characters may need to be added. It would be ideal if the interactive script would allow the user to make edits to the alphabet that is found by choosing the most frequent characters. -
Filter the data used in the training, validation and test sets by deleting every character that is not in the relevant alphabet.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the preprocess script and reading how config.yml identifies source and target data. Trace preprocessing for the training, validation, and test sets, then determine how character_filtering.yml should be loaded and generated interactively. Done means the configured alphabets can be created or edited and out-of-alphabet characters are removed from all three dataset splits.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100