Improve support for diagnostics of failed compilation: flag to preserve compilation source packages and logs

Is your feature request related to a problem? Please describe.

As a bosh operator,
In order to diagnose root cause of bosh release compilation
I need access compilation environment after the compilation failed (such as compilation log files)
And I need to be able to retry executing the failing package compilation command, possibly with additional debugging flags

Currently, after enabling the director.debug.keep_unreachable_vms property, I'm able to ssh into the compilation vm, but the bosh agent quickly removes the compilation data as observed in the following sample trace:

/var/vcap/bosh/log# less current 
[...]
2023-12-07_13:19:37.48530 [File System] 2023/12/07 13:19:37 DEBUG - Remove all /var/vcap/data/compile/galera

Describe the solution you'd like

A flag in the bosh director instructing to skip cleaning the file system on compilation failure. This flag would potentially have the side effect of preventing reuse of this faulty compilation (as its file system might fill up, preventing new compilation jobs from properly succeeding)

Describe alternatives you've considered

The current workaround is to race with the bosh agent and make a file system copy of the /var/vcap/data/compile/\<package\> directory before the compilation job ends, and then work on the copy to perform compilation diagnostics and retries.

See full diagnostic example into https://github.com/orange-cloudfoundry/paas-templates/issues/2209

Additional context

The bosh agent seems to currently unconditionally perform the clean up upon compilation completion. See likely related sources below

Clean up of the whole compilation directory
https://github.com/cloudfoundry/bosh-agent/blob/efdd50448fc8936e68a53ff5ff35c7df3d7e385c/agent/compiler/concrete_compiler.go#L80-L85

after the packaging script completion https://github.com/cloudfoundry/bosh-agent/blob/efdd50448fc8936e68a53ff5ff35c7df3d7e385c/agent/compiler/concrete_compiler.go#L109-L113

Start with the cleanup paths in bosh-agent's agent/compiler/concrete_compiler.go, especially the referenced lines around whole-directory cleanup and cleanup after packaging completion. Trace how a director flag could reach this behavior, then verify that failed compilation sources and logs remain available for diagnosis and retry without affecting unrelated compilation jobs.