Record checkpoint throughout the workflow instead of only on error

**What would you like to be added**:
We want the checkpoint file to be populated as tasks succeed instead of only at the end when an error occurs.

**Why is this needed**:
This would allow the user to use the checkpoint file to resume their cluster operations even if they terminate the process by using something like `Ctrl+C`

The issue does not name a file, test, or entry point. Locate the checkpoint handling and workflow task-completion paths, then trace how interruption and resume currently work; done means the checkpoint reflects successful tasks before an error or Ctrl+C and supports resuming cluster operations.

Written by the indexing model from the issue text.