sillsdev / sillsdev/silnlp

Improvements to extract_flex script

Open
#822 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

pipeline 2: extract
Dominant language
Python
Stars
39
Forks
7
Avg merge
1d 9h
Merged PRs (30d)
5

Description

From @mmartin9684-sil:

  • The script was failing because of UTF encoding. Opening the FLEx extract with "encoding='utf-8-sig'" fixed the problem. I'd guess that the output files should use the same encoding?
  • The extract files didn't have the same number of lines in them. The root causes were in the FLEx dataset itself. In some cases, it was issues that could be corrected in the dataset. For instance,a "txt" items that didn't have the correct language code - most entries used "dev" for the "txt" items, but some mistakenly used "en". There was a similar issue with incorrect language codes on the "gls" entries - most entries used "en" for the "gls" items, but some mistakenly used "dev". In most cases, this mean that there were multiple items in a phrase that had the same language code, so one extract file would get multiple lines while the other would get a single blank line ("\n"). I guess FLEx doesn't check for this as an error. I needed to put a check for this into the extract script. It would be nice to have the ability to specify the expected language code for the "txt" entries and the expected language code(s) for the "gls" entries, and do some checking around this.
  • There were some phrases that didn't have glosses. These resulted in blank lines in the extract file. (Phrases that used the wrong language code on the gloss also created blank lines in the extract file.) It would be nice to be able to suppress these blank lines.
  • There are a number of glosses with "???" for the gloss (or part of the gloss). I don't imagine there's a status indicator on the FLEx phrases, is there? Something like "in-progress" vs "complete" that we could look at, and only extract "complete" phrases?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the extract_flex script and reviewing how it opens FLEx extracts, assigns language codes, handles missing glosses, and processes ??? values. Clarify which requested behaviors are supported by the FLEx data before changing the script. Done should include UTF-8-SIG handling, configurable language checks, an option to suppress blank lines, and a decision on filtering incomplete phrases.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.