aimclub / aimclub/FEDOT

Add regression tests on quality & performance

未关闭
#1,028 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
enhancement test
主要语言
Python
星标
712
派生
96
平均合并
3 天 1 小时
30 天内合并 PR
10

描述

Aim is having control experiments with known result metrics for:
- autotesting regression in performance
- evaluating FEDOT after new features on these examples

First part of the issue is to write general AutoML tests (or adapt existing tests). Currently we have several integration tests that are close to such regression tests.
- [ ] Modify `test_real_cases.py` to use real expected metrics or baselines.
- [ ] Test AutoML quality on all kinds of tasks (classification, regression, forecasting)
- [x] Test on multi objective opt: must improve on all metrics -- done in `test_quality_improvement.py`

Second part of the issue is writing new tests for important features:
- [ ] Tuning: optimization with tuning must be better than just optimization)
- [ ] Preprocessing must handle some bad dirty dataset (maybe there is a test for that)
- [ ] Test that adaptive evo scheme converges faster than standard scheme (populational)

Third part is
- [ ] Test on comparisons with other AutoMLs -- FEDOT must return at least comparable quality or better (see and enhance `test_automl.py`)

This issue can be done in several PRs.
Regression tests must be put in `tests/intergration/`.

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。