Support cleaning up segments when a batch build fails

Currently if segment generation fails for one of the input files, segments created from previous input files will be left in the output directory. A new `cleanupOnFailure` flag (default is false) would trigger the deletion of these segments. Note that clearing the output directory isn't acceptable, as there might be existing files.

This change would have to be applied to all batch segment generation plugins (stand-alone, Hadoop, Spark).