Possible speed improvement

Description

I recently noticed possible speed improvement when porting segmentation example to code-generator.

According to the docs, examples, helper functions, and tests, we are calling model.train() or model.eval() inside the process function given to the Engine.

And as per this line, I think model.train() or model.eval() gets called every iteration which I think not necessary.

https://github.com/pytorch/ignite/blob/0d21a0bfe0bb03980055f9748b827cf09a6faee3/ignite/engine/engine.py#L853

To my knowledge, .train() or .eval() is only needed when changing the context (e.g. training or evaluation).
And in pytorch example, model.train() or model.eval() gets called only once when changing context (training -> evaluation, or vice versa)

Possible solution

I think we can address in the ongoing High level API development or address in the refactor of Engine design.

Workaround

For now, we can workaround by calling model.train() before every training epoch and model.eval() before every evaluation epoch.

Thanks!

Start at ignite/engine/engine.py#L853 and trace how the Engine process function invokes model.train() or model.eval() each iteration. Compare the segmentation example, code-generator work, documented examples, helper functions, tests, and linked PyTorch VAE example; done means avoiding unnecessary per-iteration mode changes without breaking training or evaluation behavior.