instadeepai / instadeepai/Mava

Evaluation uses params before training update

Open
#1,204 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
935
Forks
123
Avg merge
6d 8h
Merged PRs (30d)
9

Description

Looking at the training loop in `rec_mappo.py` (and other algorithms):

```python
learner_output = learn(learner_state)

# Evaluation uses learner_state (input), not learner_output.learner_state(output)
trained_params = unreplicate_batch_dim(learner_state.params.actor_params)
eval_metrics = evaluator(trained_params, ...)

learner_state = learner_output.learner_state # Update for next iteration
```

The evaluation runs on `learner_state.params` (the params before `learn()` was called), not `learner_output.learner_state.params` (the params after training).

This means:
- First eval uses random/initial params (no training has happened yet)
- Every eval is "one step behind" the actual trained params
- Final eval doesn't reflect the last training update (unless absolute metrics is enabled)

Is this intentional? It seems like we'd want to evaluate the params we just trained, not the ones from the previous iteration.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.