apertium / apertium/WikiExtractor

Double empty line between articles.

Open
#2 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
28
Forks
18
PR merge metrics
No merged PRs in 30d

Description

It does works, however, beside being horribly slow; it has a major issue.

It should separate articles with two empty lines, which it does, sometimes.

`\n\n`

but NOT on all articles.

`\n`

This makes the text output impossible to parse on a large scale, since no consistency is taken, it can happen everywhere.

And I am sure peoples had spent a lot of time figuring it out, as I did, and have end-up to abandon their project completly.

:/

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the extractor on a Wikipedia dump and inspect output around article boundaries. Trace the formatting path that emits article separators and compare cases showing `\n\n` with those showing `\n`. Done means article boundaries use a consistent separator throughout the output so it can be parsed reliably.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.