sillsdev / sillsdev/silnlp

Preprocessing should probably fail with an error if there are unknown parameters in the config file.

Open
#72 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

needs to be revisited pipeline 3: preprocess
Dominant language
Python
Stars
39
Forks
7
Avg merge
1d 9h
Merged PRs (30d)
5

Description

Unknown parameters in the config file seems to be silently ignored. This could lead to results which are incorrectly interpreted since the user is expecting the parameter to have an effect. For example this config file should fail at unknown_item:

  parent: en-es-MultiCCAligned_AE
  parent_use_vocab: true
  corpus_pairs:
  - type: train
    src: nhx-nhx_es_names
    trg: es-nhx_es_names
    is_lexical_data: true
  - type: train
    src: nhx-es_lexemes
    trg: es-nhx_glosses
    unknown_item: true
  - type: train,val,test
    src: nhx-nhx_2021_08_18_clean
    trg: es-nhxRT_2021_08_18_clean
    val_size: 250
    test_size: 250
  seed: 111
  share_vocab: false
  src_casing: lower
  src_vocab_size: 11682
  trg_casing: preserve
  trg_vocab_size: 32000
eval:
  multi_ref_eval: false
  steps: 1000
  early_stopping:
    metric: bleu
    min_improvement: 0.2
    steps: 4
params:
  coverage_penalty: 0.2
  word_dropout: 0.0
train:
  keep_checkpoint_max: 1
  save_checkpoints_steps: 1000```

It would be ideal to test the syntax and content of a config file prior to attempting preprocessing and describe any problems to the user.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing preprocessing with the supplied configuration and trace the config-loading and validation entry points. Done means an unknown parameter such as unknown_item causes a clear error before preprocessing begins, while valid configurations continue to work.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.