Improve SLURM executor efficiency by supporting the SLURM Apache Kafka job completion plugin

New feature

Nextflow can be quite aggressive with job schedulers. While there are good features now to rate limit the polling of the job scheduler, that does slow things down, and still doesn't really solve the problem of repeated polling of the scheduler for job status.

SLURM now supports a kafka plugin (https://slurm.schedmd.com/jobcomp_kafka.html).

Use case

Polling of SLURM by Nextflow could be almost eliminated, and delays due to polling intervals eliminated, by updating the SLURM executor to support the kafka plugin.

Suggested implementation

An event-driven executor like this is probably a significant re-write, so I realise in practice it might be simpler to implement it as a completely separate executor, rather than an option to the existing slurm executor.

I haven't looked at the code, but if the AWS Batch executor already uses EventBridge or something like that to notice job completion in AWS Batch, that mechanism could be used as a model, since the concept is the same.

Start by reading the SLURM executor and comparing it with the AWS Batch executor's event-notification approach. Consult the SLURM Apache Kafka job completion plugin documentation, then define how job-completion events would replace polling and how the new executor or option should be validated.