weecology / weecology/DeepForest
adding a way to assess model robustness
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 774
- Forks
- 265
- PR merge metrics
- No merged PRs in 30d
Description
Please describe a feature you would like to be added.
I'd like a way to run the same training scheme numerous times in order to assess the model's robustness and get an understanding of the uncertainty within the model.
Describe the solution you'd like
I'd like a parameter that can be set to train the model X amount of times and have it return a mean and standard deviation of the evaluation metrics.
Describe alternatives you've considered
Initially I restarted the kernel within one Jupyter Notebook script and ran the training multiple times while hand noting the evaluation values. Then when I tried to create a new model and rerun the training within a for loop instead of restarting the kernel, I would receive a "Weakly referenced object no longer exists" error message despite creating a new trainer.
Additional context
This issue was addressed in the DeepForest FAQ documentation. A different naming for that section would make it more clear that some code bug issues reside there.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the trainer used by the Jupyter Notebook workflow and read the DeepForest FAQ section referenced in the issue. Confirm how repeated training and evaluation are currently handled; done means a supported way to repeat training and return mean and standard deviation while addressing the reported weak-reference error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- jupyter-notebook, python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100