aws / aws/fmeval

[Feature] Support string output path in `EvalAlgorithmInterface.evaluate(save="...")`

Open
#165 3 comments 2 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
291
Forks
60
PR merge metrics
No merged PRs in 30d

Description

Today the `save` parameter of `EvalAlgorithmInterface.evaluate()` is [just a boolean](https://github.com/aws/fmeval/blob/220fdc090ee4f1db7174f6e7661a282a5987d1af/src/fmeval/eval_algorithms/eval_algorithm.py#L35).

From hunting around I found that the output path is [taken](https://github.com/aws/fmeval/blob/220fdc090ee4f1db7174f6e7661a282a5987d1af/src/fmeval/util.py#L47) either from an environment variable or else a [default under /tmp](https://github.com/aws/fmeval/blob/220fdc090ee4f1db7174f6e7661a282a5987d1af/src/fmeval/constants.py#L4)... And that it should also be overrideable by setting the obviously-supposed-to-be-private property `eval_algo._eval_results_path`.

IMO it's harder and less obvious than it should be for a developer using the library to save results in a folder they want. It'd be much easier if we could support `eval_algo.evaluate(save="my/cool/folder")` and ideally automatically create the provided folder if it doesn't already exist?

Contributor guide

Open the contributing guide

Research direction

Start with EvalAlgorithmInterface.evaluate() in src/fmeval/eval_algorithms/eval_algorithm.py, then trace the output-path handling in src/fmeval/util.py and the default in src/fmeval/constants.py. Confirm that a string passed to save selects the requested folder, creates it when needed, and preserves the existing behavior for boolean values.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.