WikiExtractor / WikiExtractor/wikiextractor

[Feature Request]: Capture Paragaph Heading information

Open
#268 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
4k
Forks
1k
Avg merge
1m
Merged PRs (30d)
10

Description

If we were able to split paragraphs by double newline characters, and somehow tag a paragraph heading to be destinct from a paragraph, then we could retain useful paragraph heading information for NLP tasks.

e.g. often there are paragraph headings such as:

  • Aftermath and cause
  • History and infrastructure
  • Accomplishment
  • Mission

They could all exist across multiple articles. If we could tag a paragraph heading with a symbol in the output, potentially this oculd be useful in doc2vev tags for example.

e.g of output of dummy text assuming # as the symbol to tag paragraph headings

Here is a sentence. Here is a second sentence. This paragraph doesn't have a paragraph heading.

#History and infrastructure
Here is a second paragraph. It had a paragraph heading and can be tagged as such. Sentences from other articles could likely also have the same paragraph heading. It is useful to capture this information.
Here is a third paragraph. It is also attributed to the same paragraph heading as the paragraph before.

Here is another paragraph without paragraph heading.

#Flight
Another paragraph, this time to do with flights. Just a stupid example

If text was output in this format we could split paragraphs by single newline character and groups of paragraphs by double newline character. We could also introduce a tag to associate with the one or multiple paragraphs that fall under such a paragraph.

Open to any other suggestions. If the '#' symbol was to be used, we would need to escape any text that includes in at the start (but I am unsure that is even allowed, so not a big issue... worth covering as an edge case anyway).

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names no files, tests, or entry points, so first trace the existing paragraph and newline handling in the extractor. Define how headings and their associated paragraphs should be represented, including heading-like text at paragraph starts, then add coverage showing the proposed output and edge cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
content
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.