globalwordnet / globalwordnet/semcor

Dateline/byline text missing entirely from press_reportage articles

Open
#32 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
0
Forks
0
Avg merge
2d 22h
Merged PRs (30d)
24

Description

Part of #5. Found as a side-effect of #15's investigation (comparing this
corpus's `text` against `nltk.corpus.brown` for the thousands-separator
comma fix); not a comma bug itself, but the same diff run surfaced it.

A number of `press_reportage` files are missing a whole dateline/byline
string from `text` that Brown's plaintext still has, e.g. (aggregated, not
verbatim excerpts):

- A file-opening `,--` or `,.--` dateline
(`Austin,Texas--`-shaped, `Washington,Feb.9--`-shaped, `Miami,Fla.,March17--`-shaped,
`Annapolis,Jan.7--`-shaped) is entirely absent from the corresponding
sentence's `text` in this corpus, appearing 6+ times just among the files
#15's scan happened to touch.
- Similarly, a governor/official's full name-and-title byline fragment
(`,Jr.,`-shaped) is missing entirely in at least one file.

This is a bigger loss than a stray character -- a whole leading phrase of
the sentence is gone, not just reformatted. Worth checking whether this is
systematic across all `press_reportage` files (this was only noticed
incidentally on the subset #15's scan happened to diff) and, if so, whether
it's recoverable the same way as the rest of this corpus's Brown-alignment
fixes (#8-#15) or whether it reflects a deliberate editorial choice
upstream to drop datelines.

### Suggested next step

A dedicated scan across all `press_reportage` files comparing each file's
opening sentence against Brown's corresponding opening would confirm scope
before deciding on a fix.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with all press_reportage files and compare each opening sentence with the corresponding opening in nltk.corpus.brown, extending the subset examined by issue #15. Record which dateline or byline fragments are absent and determine whether the differences are recoverable alignment errors or deliberate editorial choices. Done means the scope is documented and a clear fix direction is established.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.