WikiExtractor / WikiExtractor/wikiextractor
Bad xml parsing when closing tags are omitted
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4k
- Forks
- 1k
- Avg merge
- 1m
- Merged PRs (30d)
- 10
Description
The wikipedia page:
https://en.wikipedia.org/wiki/Jeff_Beck
In the latest dump file contains the following text:
;Collaborative albums
*''[[Beck, Bogert & Appice (album)|Beck, Bogert & Appice]]'' (1973) <small>(by [[Beck, Bogert & Appice]])</small>
*''[[Frankie's House (soundtrack album)|Frankie's House]]'' (1992) <small>(with Jed Leiber)</small>
*''[[Barabajagal (album)|Donovan and Jeff Beck Group]]'' (1970) <small>
As we can see on each of the lines we have a <small>...</small> tag, but for some weird reason, the small tag on the last line doesn't have a closing tag.
The output of wikiextractor is
- Collaborative albums
- "<a href="Beck%2C%20Bogert%20%26amp%3B%20Appice%20%28album%29">Beck, Bogert & Appice</a>" (1973)
- "<a href="Frankie%27s%20House%20%28soundtrack%20album%29">Frankie's House</a>" (1992)
- "<a href="Barabajagal%20%28album%29">Donovan and Jeff Beck Group</a>" (1970) <small>
Notice the <small> in the end.
In HTML, some elements might work well, even with a missing closing tag:
<p>This is a paragraph.
<br>
In XML, it is illegal to omit the closing tag. All elements must have a closing tag:
<p>This is a paragraph.</p>
<br />
So the first bug is that the wikiextractor didn't change the tag to be <small/>
The second bug is that the <small> was not omitted - creating an improper XML.
I think that by first - adding closing tags and creating a valid XML and then running the rest of the parsing process we will overcome this bug and other similar bugs.
Food for thought: when do we close an open tag? <small> should be closed with </small> when we encounter the next closing tag which is not </small> but <br> should be closed immediately as </br>.
Sam.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the reported output from the Jeff Beck Wikipedia page or its dump and trace the XML parsing path in wikiextractor. Compare handling of omitted closing tags with the examples in the issue; done means malformed input no longer leaves an unclosed tag in the generated XML while normal extraction still works.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100