Azure / Azure/azureml-examples

How to add a custom metric to text generation pipeline + enable MLflow tracking

Open
#2,880 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Jupyter Notebook
Stars
2k
Forks
1.7k
Avg merge
18h 18m
Merged PRs (30d)
2

Description

### Describe your suggestion

I am following [this ](https://github.com/Azure/azureml-examples/blob/main/cli/foundation-models/system/finetune/text-generation/text-generation-pipeline.yml) example of a text generation pipeline for finetuning.

I see from the [yaml api](https://github.com/Azure/azureml-assets/blob/main/assets/training/finetune_acft_hf_nlp/components/pipeline_components/text_generation/README.md) that there is an evaluation config parameters argument that includes computing metrics.
![image](https://github.com/Azure/azureml-examples/assets/13874772/8f0ccc0a-865f-42a5-8f5e-bbe0a06928b5)

However, it's not clear how to structure this argument. If I have my own custom compute_metrics function, how do I structure the arguments of the pipeline to utilize it? (i.e. the argument for `evaluation_config` is uri_file type, while it's a json serialized string for the `evaluation_config_params` argument).

An additional question, is there a way to turn on mlflow tracking with this pipeline so that the metrics get logged?

### Additional details

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the text-generation pipeline example in cli/foundation-models/system/finetune/text-generation/text-generation-pipeline.yml and the evaluation configuration README linked in the issue. Clarify how evaluation_config and evaluation_config_params represent a custom compute_metrics function, and determine whether MLflow tracking is supported. Done means the example or documentation answers both usage questions with a reproducible configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure
Domain
cloud, documentation, machine-learning
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.