allenai / allenai/tango

Hyperparameter sweeps

Ouverte
#143 1 commentaire 2 réactions 0 personnes assignées Voir sur GitHub
feature request
Langage dominant
Python
Étoiles
572
Forks
55
Métriques de merge des PR
Aucune PR mergée en 30 j

Description

It would be great if Tango provided a simple yet general mechanism for doing hyperparameter searches. Here's an outline of how that could look 👇

We provide a new subcommand: `tango sweep`. This command takes
- a "sweep" config,
- a regular experiment config,
- and a target step name.

For example:

```
tango sweep sweep-config.jsonnet target-config.jsonnet step-name
```

`step-name` should correspond to the main step of interest in `target-config.jsonnet` that provides the results we are trying to optimize for (might require #142). For example, this could be a validation/eval step that spits out some metrics of your model on a dataset.

The `sweep-config.jsonnet` would define which hyperparameters to search and how to search over them. By "hyperparemeters" I really just mean any fields in `target-config.jsonnet`. There are many ways we could do the search, and this is an active area of research. So I think it would be ideal if we were able to integrate with existing hyperparameter sweep frameworks / platforms, like [W&B](https://docs.wandb.ai/guides/sweeps), [Optuna](https://optuna.org/), etc. These integrations should be optional, however, and I think we should provide a simple default search method, which could just be grid search.

Under the hood `tango sweep` could use the `tango run` subcommand with the `--overrides` parameter to select hyperparameter values. We should also be able to run the search in parallel.

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.