Notes contain non-visible characters
Nobody has claimed this yet.
- Dominant language
- Jupyter Notebook
- Stars
- 3.4k
- Forks
- 1.7k
- PR merge metrics
- No merged PRs in 30d
Description
There are three non-visible characters (\x7f, \x14, and \x13) that appear in the notes. These characters cause issues when working with many text processing tools. Removing these characters will make it easier to work with the notes and should not change the underlying meaning.
Currently, when using processing tools (e.g., cTAKES) that process a directory, one can workaround this issue by manually removing the offending characters:
find notes -type f -exec sed -i 's/\x7f//g; s/\x14//g; s/\x13//g' {} +
These characters appear in ~700 notes. A truncated character distribution shows:
[ # ...
('\x13', 677),
('\x14', 49),
('\x7f', 2)]
There does not appear to be overlap among the affected notes.
find notes -type f | xargs grep --color='auto' -P -l "[\x7f]" | sort | uniq -c
1 notes/1062359
1 notes/862440
find notes -type f | xargs grep --color='auto' -P -l "[\x14]" | sort | uniq -c
1 notes/1437693
1 notes/1482241
...
find notes -type f | xargs grep --color='auto' -P -l "[\x13]" | sort | uniq -c
1 notes/1901399
1 notes/1902291
...
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the affected files under notes/ and the provided find, grep, and sed commands. Verify that \x7f, \x14, and \x13 are removed from the affected notes without changing other content, and confirm the notes can be processed without the manual workaround.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- shell
- Domain
- data
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 38/100