MIT-LCP / MIT-LCP/mimic-code

Notes contain non-visible characters

Open
#137 2 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

mimic-iii
Dominant language
Jupyter Notebook
Stars
3.4k
Forks
1.7k
PR merge metrics
No merged PRs in 30d

Description

There are three non-visible characters (\x7f, \x14, and \x13) that appear in the notes. These characters cause issues when working with many text processing tools. Removing these characters will make it easier to work with the notes and should not change the underlying meaning.

Currently, when using processing tools (e.g., cTAKES) that process a directory, one can workaround this issue by manually removing the offending characters:

find notes -type f -exec sed -i 's/\x7f//g; s/\x14//g; s/\x13//g' {} +

These characters appear in ~700 notes. A truncated character distribution shows:

[ # ...
 ('\x13', 677),
 ('\x14', 49),
 ('\x7f', 2)]

There does not appear to be overlap among the affected notes.

find notes -type f | xargs grep --color='auto' -P -l "[\x7f]" | sort | uniq -c
      1 notes/1062359
      1 notes/862440
find notes -type f | xargs grep --color='auto' -P -l "[\x14]" | sort | uniq -c
      1 notes/1437693
      1 notes/1482241
...
find notes -type f | xargs grep --color='auto' -P -l "[\x13]" | sort | uniq -c
      1 notes/1901399
      1 notes/1902291
...

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the affected files under notes/ and the provided find, grep, and sed commands. Verify that \x7f, \x14, and \x13 are removed from the affected notes without changing other content, and confirm the notes can be processed without the manual workaround.

Written by the indexing model from the issue text.

Assessment

Tech stack
shell
Domain
data
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.