Support for per-rule spot instances?

Currently, if we want to use spot instances with Batch, we need to send jobs to a queue that is connected to an environment set up using spot instances. But in reality, it often makes sense to send some long-running jobs to on-demand instances while others to spot instances, to minimize chances of interuptions. As far as I can undertand, this is currently not possible?

I'm wondering what would be the best way to bring that support. Ideally, we want a directive spot: True in the rule definition, but this requires modifying Snakemake code.

Alternatively, we can add a command line argument --spot-rules that takes a comma separated string of rule names that should be sent to spot instances (and --spot-job-queue that specifies the job queue to use for those jobs). The argument to --spot-rules can be quite long and unmanageable but that's the kind of setting that makes sense in a snakemake profile config so maybe it's acceptable?

Start by reviewing how Snakemake rules are represented and how AWS Batch job queues are selected. Compare the proposed per-rule spot: True directive with the --spot-rules and --spot-job-queue options, and define the desired queue-selection behavior before implementation.