Save debug information for a smaller reproducer if thunder.jit fails

🚀 Feature

In case, we hit a failure during thunder.jit invocation, it would be great to have a debug option where thunder.jit would save the fn to jit, args, kwargs (and thunder.jit arguments) it received so that we can reproduce the failure in a smaller script for nicer debugging experience.

Motivation

With thunderFX path, we may have multiple invocations of thunder.jit with different sub-regions of the model and inputs. It may happen that one of the thunder.jit invocation may fail. In this case, it would be great if we can save some debug information so that it can be reproduced independently with a smaller script.

NOTE - This maybe helpful outside of thunderFX path as well where there maybe a lot of boilerplate training code around thunder.jit and enabling this option will dump a smaller repro if thunder.jit invocation fails.

Alternatives

Manually insert points where the required details are captured (but this requires some knowledge of the codebase).

Additional context

Related - https://github.com/Lightning-AI/lightning-thunder/issues/270, https://github.com/Lightning-AI/lightning-thunder/issues/387

cc: @riccardofelluga (who introduced similar feature for nvFuser region debugging #387) for ideas and suggestions.

cc @carmocca @apaz-cli