Add retry when NetMHCstabpan fails

Nobody has claimed this yet.

Sometimes NetMHCstabpan fails in the following way but restarts of the pipeline will succeed:

Traceback (most recent call last):
  File "/usr/local/bin/pvacseq", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/pvactools/tools/pvacseq/main.py", line 116, in main
    args[0].func.main(args[1])
  File "/usr/local/lib/python3.7/site-packages/pvactools/tools/pvacseq/run.py", line 132, in main
    pipeline.execute()
  File "/usr/local/lib/python3.7/site-packages/pvactools/lib/pipeline.py", line 506, in execute
    PostProcessor(**post_processing_params).execute()
  File "/usr/local/lib/python3.7/site-packages/pvactools/lib/post_processor.py", line 38, in execute
    self.call_netmhc_stab()
  File "/usr/local/lib/python3.7/site-packages/pvactools/lib/post_processor.py", line 121, in call_netmhc_stab
    NetMHCStab(self.net_chop_fh.name, self.netmhc_stab_fh.name, self.file_type, self.top_score_metric).execute()
  File "/usr/local/lib/python3.7/site-packages/pvactools/lib/netmhc_stab.py", line 86, in execute
    raise Exception("NetMHCstabpan encountered an error during processing.\n{}".format(response.content.decode()))
Exception: NetMHCstabpan encountered an error during processing.
<html><head><title>635A40E9000078ABEBEE3CAB failed</title></head>
<body>
<H1>Failed run</H1>
<P>Unfortunately, your job failed. It responded with this:</P>
<PRE>
Content-Type: text/html; charset=ISO-8859-1

<BODY LINK="#0000ff" BGCOLOR="#ffffff">

<script language=JavaScript type = "text/javascript">
  var loc = window.top.location.search.substring(1);
  if (loc != null && loc !== "" && loc.length > 1) {
      document.write('<center><h2>',loc,' Server Output - DTU Health Tech</h2></center>');}
  </script> 
<hr>
<pre>
</pre>
<hr>
Go <a href="javascript:history.back()"><b>back</b></a>.
<!--
<button type="button"
onclick="window.top.location.reload(true);")>
New Submission</button>
-->
</BODY>
</HTML>


</PRE>

This may indicate that a retry (after some appropriate amount of wait) for this error condition would succeed. We should also do the same for NetChop

Start with pvactools/lib/netmhc_stab.py and its execute path shown in the traceback, then inspect the NetChop call in the post-processing pipeline. Determine how this transient failure is identified and what wait interval is appropriate; done means the pipeline retries failed NetMHCstabpan and NetChop jobs instead of requiring a restart.

Written by the indexing model from the issue text.

Description

Contributor guide

First steps

Research direction

Assessment