Azure / Azure/azureml-examples
How to add a custom metric to text generation pipeline + enable MLflow tracking
- Dominant language
- Jupyter Notebook
- Stars
- 2k
- Forks
- 1.7k
- Avg merge
- 18h 18m
- Merged PRs (30d)
- 2
Description
### Describe your suggestion
I am following [this ](https://github.com/Azure/azureml-examples/blob/main/cli/foundation-models/system/finetune/text-generation/text-generation-pipeline.yml) example of a text generation pipeline for finetuning.
I see from the [yaml api](https://github.com/Azure/azureml-assets/blob/main/assets/training/finetune_acft_hf_nlp/components/pipeline_components/text_generation/README.md) that there is an evaluation config parameters argument that includes computing metrics.

However, it's not clear how to structure this argument. If I have my own custom compute_metrics function, how do I structure the arguments of the pipeline to utilize it? (i.e. the argument for `evaluation_config` is uri_file type, while it's a json serialized string for the `evaluation_config_params` argument).
An additional question, is there a way to turn on mlflow tracking with this pipeline so that the metrics get logged?
### Additional details
_No response_
Contributor guide
Research direction
Start with the text-generation pipeline example in cli/foundation-models/system/finetune/text-generation/text-generation-pipeline.yml and the evaluation configuration README linked in the issue. Clarify how evaluation_config and evaluation_config_params represent a custom compute_metrics function, and determine whether MLflow tracking is supported. Done means the example or documentation answers both usage questions with a reproducible configuration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure
- Domain
- cloud, documentation, machine-learning
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100