pytorch / pytorch/ignite

Epoch is implicitly incremented if terminated on iteration

Open
#1,386 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug module: engine
Dominant language
Python
Stars
4.8k
Forks
726
Avg merge
5d 21h
Merged PRs (30d)
5

Description

🐛 Bug description

Below code shows the error:

from ignite.engine import Engine, Events
from ignite.utils import setup_logger

stop_iter = 50
epoch_length = 100
max_epochs = 2

trainer = Engine(lambda e, b: print(b, end=" "))
trainer.logger = setup_logger("trainer")
state = trainer.state

@trainer.on(Events.ITERATION_COMPLETED(every=stop_iter))
def stop():
    print("--> stop at {}".format(trainer.state.iteration))
    trainer.terminate()

data = list(range(epoch_length))

print("- Start from {} iteration".format(state.iteration))
state = trainer.run(data, max_epochs=max_epochs, epoch_length=epoch_length)
print("- Ended on {} iteration | {} epoch".format(state.iteration, state.epoch))

print("-- Do something else")

print("- Continue from {} iteration".format(state.iteration))
state = trainer.run(data, max_epochs=max_epochs, epoch_length=epoch_length)
print("- Ended on {} iteration | {} epoch".format(state.iteration, state.epoch))

print("-- Do something else")

print("- Continue from {} iteration".format(state.iteration))
state = trainer.run(data, max_epochs=max_epochs, epoch_length=epoch_length)
print("- Ended on {} iteration | {} epoch".format(state.iteration, state.epoch))

print("-- Do something else")

print("- Continue from {} iteration".format(state.iteration))
state = trainer.run(data, max_epochs=max_epochs, epoch_length=epoch_length)
print("- Ended on {} iteration | {} epoch".format(state.iteration, state.epoch))

print("-- Do something else")

print("- Continue from {} iteration".format(state.iteration))
state = trainer.run(data, max_epochs=max_epochs, epoch_length=epoch_length)
print("- Ended on {} iteration | {} epoch".format(state.iteration, state.epoch))

The issue is that iteration and epoch start to be unrelated which is a bug.

Environment

  • PyTorch Version (e.g., 1.4):
  • Ignite Version (e.g., 0.3.0):
  • OS (e.g., Linux):
  • How you installed Ignite (conda, pip, source):
  • Python version:
  • Any other relevant information:

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the reproduction in the issue, then inspect the Engine.run flow and the trainer.terminate handling. Trace how iteration and epoch are updated when termination occurs at iteration 50. Done means repeated runs preserve a consistent relationship between the iteration and epoch values shown in the reproduction.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.