weecology / weecology/DeepForest

adding a way to assess model robustness

Open
#533 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Feature Request
Dominant language
Python
Stars
774
Forks
265
PR merge metrics
No merged PRs in 30d

Description

Please describe a feature you would like to be added.
I'd like a way to run the same training scheme numerous times in order to assess the model's robustness and get an understanding of the uncertainty within the model.

Describe the solution you'd like
I'd like a parameter that can be set to train the model X amount of times and have it return a mean and standard deviation of the evaluation metrics.

Describe alternatives you've considered
Initially I restarted the kernel within one Jupyter Notebook script and ran the training multiple times while hand noting the evaluation values. Then when I tried to create a new model and rerun the training within a for loop instead of restarting the kernel, I would receive a "Weakly referenced object no longer exists" error message despite creating a new trainer.

Additional context
This issue was addressed in the DeepForest FAQ documentation. A different naming for that section would make it more clear that some code bug issues reside there.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the trainer used by the Jupyter Notebook workflow and read the DeepForest FAQ section referenced in the issue. Confirm how repeated training and evaluation are currently handled; done means a supported way to repeat training and return mean and standard deviation while addressing the reported weak-reference error.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter-notebook, python
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.