sillsdev / sillsdev/silnlp

Parameterize the folder structure for the pipeline, i.e. remove the hard coded folder structure.

Open
#101 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement needs to be revisited
Dominant language
Python
Stars
39
Forks
7
Avg merge
1d 9h
Merged PRs (30d)
5

Description

The hard coded folder structure is making it difficult to deal with extracts from multiple sources.
When uses extract_corpora to extract from eBible and Paratext data something like the following folder structure is necessary because the folders are hard-coded:

eBible
   -   MT
       -    corpora
       -    experiments
       -    scripture
       -    terms
   -     Paratext
       -    projects
Paratext
   -   MT
       -    corpora
       -    experiments
       -    scripture
       -    terms
   -     Paratext
       -    projects

Could we use a configuration file to specify various folders and have a command line option or an environment variable in order to set which folder config file should be used? The following folders are those which might be used, though the pipeline might not be creating all of the data (yet). Some filtering from unzipped_projects to processed_projects may use other tools.
A projects_for_extraction parameter would explicitly specify whether that is the same as the unzipped_projects folder or one of those on which further processing had been done.

We could also make a change that would allow the training code to look in any arbitary folder for other corpus data. Since these corpora are large moving them to match our hard-coded folder structure can be time consuming. These are some of the folder functions that we may want to configure. For example we may want to distinguish between raw unzipped projects and projects ready for extraction into an aligned file.

zipped_projects 
unzipped_projects
projects_for_extraction
extracted_scripture_files
extracted_files_metadata
extracted_terms
scripture_files1_for_training
scripture_files2_for_training
corpus_files1_for_training
corpus_files2_for_training

We might want to use these in named sets. If this is implemented, then we would need a way to name the set in the config file, and a way to specify the set to be used in the command line, alternatively, we could have multiple folder config files and switch between them. However named sets would allow for both sets of folders to be referenced in the training experiment config file.

folder_set_name : eBible
  - zipped_projects : 
  - unzipped_projects
  - processed_project
  etc...

folder_set_name : Paratext
  - zipped_projects : 
  - unzipped_projects
  - processed_project
  etc...

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the extract_corpora pipeline and the training experiment configuration that currently assume fixed folders. Define the configuration and selection entry points for named folder sets or arbitrary corpus locations; done means extraction and training can use multiple source layouts without moving data, with the supported behavior covered by tests or documented examples.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.