WikiExtractor / WikiExtractor/wikiextractor
Non-textual elements score and mapframe are not filtered out
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4k
- Forks
- 1k
- Avg merge
- 1m
- Merged PRs (30d)
- 10
Description
Several elements with non-textual content such as maps and musical scores (elements mapframe and score) are not filtered out. Steps to reproduce:
- Download this dump:
https://dumps.wikimedia.org/enwiki/20221020/enwiki-20221020-pages-articles1.xml-p1p41242.bz2 - Invoke the following command to list lines that contain the opening tags of these elements:
wikiextractor --no-templates --html-safe '' -o - dumps.wikimedia.org/enwiki/20221020/enwiki-20221020-pages-articles1.xml-p1p41242.bz2 | grep '<\(score\|mapframe\)\b'
Output:
<score sound="1"> % Adding least one space before each line is recommended
<mapframe latitude="37.7235" longitude="23.4819" zoom="10" width="200" height="131" align="left" />Aegina is roughly triangular in shape, approximately from east to west and from north to south, with an area of .
<score sound="1">{ \time 4/4 c'4 c' g' g' | a' a' g'2 | f'4 f' e' e' | d' d' c'2 | g'4 g' f' f' | e' e' d'2 | g'4 \times 2/3 { f'8 f' f' } e'4 d' | c' r r2 | \bar "|." } \addlyrics { A B C D E F G H I J K L M N O P Q R S T U V dub- a- U X Y "Z(ed)" }</score>
<score sound="1">
<score sound="1">
<mapframe width=400 height=200 zoom=6 latitude=42.977 longitude=-76.506 align=left text="The original path of the Erie Canal">
<mapframe latitude="37.807984" longitude="-122.475411" zoom="18" width="213" height="180" align="right">
In general, the output also contains the content delimited by the tags (musical scores and map data). In some cases, both of the opening/closing tags (or parth the score itself) for musical scores are missing, e.g. article id="152940" from dumps.wikimedia.org/enwiki/20221020/enwiki-20221020-pages-articles3.xml-p151574p311329.bz2 contains only the opening <score>:
Sheet music does not often explicitly indicate "Bebung". Composers assumed that, like other ornaments, performers would apply "bebung" at their discretion. Where sheet music does indicate "bebung", it appears as a series of dots above or below a note, with the number of dots indicating the number of finger movements. For example: <score>
Carl Philipp Emanuel Bach called the vibrato "Bebung", however other composers like Johann Mattheson had described the term earlier on. C.P.E Bach often used Bebung in his
More often, we see the whole score with the closing tag, but no opening tag.
There similar issues with other tags (#300) and table formatting (#298).
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the issue with the documented wikiextractor command and the provided dump, then trace the entry point that handles score and mapframe elements. Done means those elements and their delimited content no longer appear in the extracted output, including cases with missing closing or opening tags.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100