[Feature] Support string output path in `EvalAlgorithmInterface.evaluate(save="...")`
- Dominant language
- Python
- Stars
- 291
- Forks
- 60
- PR merge metrics
- No merged PRs in 30d
Description
Today the `save` parameter of `EvalAlgorithmInterface.evaluate()` is [just a boolean](https://github.com/aws/fmeval/blob/220fdc090ee4f1db7174f6e7661a282a5987d1af/src/fmeval/eval_algorithms/eval_algorithm.py#L35).
From hunting around I found that the output path is [taken](https://github.com/aws/fmeval/blob/220fdc090ee4f1db7174f6e7661a282a5987d1af/src/fmeval/util.py#L47) either from an environment variable or else a [default under /tmp](https://github.com/aws/fmeval/blob/220fdc090ee4f1db7174f6e7661a282a5987d1af/src/fmeval/constants.py#L4)... And that it should also be overrideable by setting the obviously-supposed-to-be-private property `eval_algo._eval_results_path`.
IMO it's harder and less obvious than it should be for a developer using the library to save results in a folder they want. It'd be much easier if we could support `eval_algo.evaluate(save="my/cool/folder")` and ideally automatically create the provided folder if it doesn't already exist?
Contributor guide
Research direction
Start with EvalAlgorithmInterface.evaluate() in src/fmeval/eval_algorithms/eval_algorithm.py, then trace the output-path handling in src/fmeval/util.py and the default in src/fmeval/constants.py. Confirm that a string passed to save selects the requested folder, creates it when needed, and preserves the existing behavior for boolean values.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 50/100