WikiExtractor / WikiExtractor/wikiextractor

Bad xml parsing when closing tags are omitted

Open
#133 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
4k
Forks
1k
Avg merge
1m
Merged PRs (30d)
10

Description

The wikipedia page:

https://en.wikipedia.org/wiki/Jeff_Beck

In the latest dump file contains the following text:

;Collaborative albums
*''[[Beck, Bogert & Appice (album)|Beck, Bogert & Appice]]'' (1973) <small>(by [[Beck, Bogert & Appice]])</small>
*''[[Frankie's House (soundtrack album)|Frankie's House]]'' (1992) <small>(with Jed Leiber)</small>
*''[[Barabajagal (album)|Donovan and Jeff Beck Group]]'' (1970) <small>

As we can see on each of the lines we have a <small>...</small> tag, but for some weird reason, the small tag on the last line doesn't have a closing tag.

The output of wikiextractor is

- Collaborative albums
- "<a href="Beck%2C%20Bogert%20%26amp%3B%20Appice%20%28album%29">Beck, Bogert & Appice</a>" (1973)
- "<a href="Frankie%27s%20House%20%28soundtrack%20album%29">Frankie's House</a>" (1992)
- "<a href="Barabajagal%20%28album%29">Donovan and Jeff Beck Group</a>" (1970) <small>

Notice the <small> in the end.

In HTML, some elements might work well, even with a missing closing tag:

<p>This is a paragraph.
<br>

In XML, it is illegal to omit the closing tag. All elements must have a closing tag:

<p>This is a paragraph.</p>
<br />

So the first bug is that the wikiextractor didn't change the tag to be <small/>
The second bug is that the <small> was not omitted - creating an improper XML.

I think that by first - adding closing tags and creating a valid XML and then running the rest of the parsing process we will overcome this bug and other similar bugs.
Food for thought: when do we close an open tag? <small> should be closed with </small> when we encounter the next closing tag which is not </small> but <br> should be closed immediately as </br>.

Sam.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the reported output from the Jeff Beck Wikipedia page or its dump and trace the XML parsing path in wikiextractor. Compare handling of omitted closing tags with the examples in the issue; done means malformed input no longer leaves an unclosed tag in the generated XML while normal extraction still works.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.