globalwordnet / globalwordnet/semcor
Dateline/byline text missing entirely from press_reportage articles
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 24
Description
Part of #5. Found as a side-effect of #15's investigation (comparing this
corpus's `text` against `nltk.corpus.brown` for the thousands-separator
comma fix); not a comma bug itself, but the same diff run surfaced it.
A number of `press_reportage` files are missing a whole dateline/byline
string from `text` that Brown's plaintext still has, e.g. (aggregated, not
verbatim excerpts):
- A file-opening `,--` or `,.--` dateline
(`Austin,Texas--`-shaped, `Washington,Feb.9--`-shaped, `Miami,Fla.,March17--`-shaped,
`Annapolis,Jan.7--`-shaped) is entirely absent from the corresponding
sentence's `text` in this corpus, appearing 6+ times just among the files
#15's scan happened to touch.
- Similarly, a governor/official's full name-and-title byline fragment
(`,Jr.,`-shaped) is missing entirely in at least one file.
This is a bigger loss than a stray character -- a whole leading phrase of
the sentence is gone, not just reformatted. Worth checking whether this is
systematic across all `press_reportage` files (this was only noticed
incidentally on the subset #15's scan happened to diff) and, if so, whether
it's recoverable the same way as the rest of this corpus's Brown-alignment
fixes (#8-#15) or whether it reflects a deliberate editorial choice
upstream to drop datelines.
### Suggested next step
A dedicated scan across all `press_reportage` files comparing each file's
opening sentence against Brown's corresponding opening would confirm scope
before deciding on a fix.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with all press_reportage files and compare each opening sentence with the corresponding opening in nltk.corpus.brown, extending the subset examined by issue #15. Record which dateline or byline fragments are absent and determine whether the differences are recoverable alignment errors or deliberate editorial choices. Done means the scope is documented and a clear fix direction is established.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100