Improved support for HTCondor

I've been tasked with trying to improve nextflow/HTCondor interoperability by the Powers That Be at UWisc (and my own pipeline's needs).

Initially, I thought the major issue would be the lack of a shared file system in HTCondor (#3697). However, I've encountered other issues that suggest a proper feature request might be in order.

1. Getting access to a node with submit capabilities. This is a HTCondor-side issue, and I'm working it out with them. It should be doable. (It's possible the solution lies in HTCondor's grid setup, but I don't know enough to say for sure.
2. Shared file system. Our setup has a shared 'staging' drive, which will serve as a shared file system while I get the other issues worked out. Eventually, I think Wave/Fusion should solve this problem.
3. Launching containerized jobs from Nextflow. Currently all HTCondor jobs are submitted with the descriptor "universe = vanilla", which forces the job to be run as "Vanilla Universe" job, ie locally on each server.
   • HTCondor has a separate "Docker" universe to run jobs in docker containers. One simply needs to specifiy 'universe=docker,docker_image=my/dockerimage:version'.
   • HTCondor also has a "Container" universe, which sets up jobs to run in Docker containers or Singularity containers. If a universe isn't specified, HTCondor will attempt to determine the universe being requested. Jobs with no image specified are run as vanilla/local jobs, jobs with a docker image url specified are run in docker containers, and jobs with a ".sif" file are run in that singularity container.
   • I think most users would benefit from the ability to run containerized jobs. Maybe if 'docker=true' or 'singularity=true', the appropriate universe could be specified along with the image file?

I will add to this post as more issues are encountered. I'm writing something similar to the HTCondor people.

Start by reviewing Nextflow's existing HTCondor submission handling and the HTCondor universe and container documentation linked in the issue. The work is complete when the supported HTCondor execution and container behavior is defined and implemented, with the relevant submission behavior verified.