Add process-tree kill and missing timeouts to simulation subprocesses

Problem

Simulation subprocesses have inconsistent timeout handling and kill strategies. A hung simulation can block a worker indefinitely or leave orphaned processes.

Gaps Identified

1. No timeout on run_bundle_gems (Critical)

File: server/app/jobs/dj_jobs/run_simulate_data_point.rb:953-955

pid = Process.spawn(install_env, cmd, [:err, :out] => [log_path, 'w'], chdir: analysis_dir)
Process.wait pid  # No timeout wrapper!

A hung bundle install blocks the worker forever.

2. UrbanOpt runner lacks process cleanup on timeout

File: server/app/jobs/dj_jobs/urban_opt.rb:137-143, 203-206
Has Timeout.timeout(28800) but no rescue Timeout::Error block to kill orphaned processes. The Timeout::Error would propagate unhandled.

3. In-server kill only handles 2 levels deep

File: server/app/jobs/dj_jobs/run_simulate_data_point.rb:220-237
Uses ps -o pid= --ppid to find one child, then SIGKILL. Does not kill grandchildren. The external runner (run_chunk.rb:177-192) uses process-group kill (Process.kill('KILL', -pid)) which is far more robust.

4. Utility::Oss.run_script kills with SIGTERM, not SIGKILL

File: server/app/lib/utility/oss.rb:117
Sends SIGTERM which may not kill hung processes. Also has a typo bug: fullpath should be full_path on line 120.

Proposed Fix

1. Add timeout to run_bundle_gems: Wrap Process.wait in Timeout.timeout(4 * 3600) with rescue Timeout::Error + kill_process_tree (matching the external runner's pattern)
2. Add process-group kill to run_simulate_data_point.rb: Adopt the pgroup: true + Process.kill('KILL', -pid) pattern from run_chunk.rb to kill the entire process tree
3. Add timeout rescue to UrbanOpt: Wrap the UO run/process Timeout.timeout calls with rescue Timeout::Error that kills spawned processes
4. Fix Utility::Oss.run_script: Change SIGTERM to SIGKILL, fix the fullpath typo
5. Standardize the kill pattern: Extract a shared kill_process_tree utility method that all runners can use

Files to Modify

• server/app/jobs/dj_jobs/run_simulate_data_point.rb
• server/app/jobs/dj_jobs/urban_opt.rb
• server/app/lib/utility/oss.rb
• Potentially a new shared utility for process-tree kill

Reference

The external batch runner (external_batch/runner/run_chunk.rb) already implements the correct pattern with process-group isolation and kill. This should be the model for the in-server runner.

Start with the process handling in server/app/jobs/dj_jobs/run_simulate_data_point.rb, especially the cited run_bundle_gems and in-server kill sections, and compare them with external_batch/runner/run_chunk.rb. Then inspect server/app/jobs/dj_jobs/urban_opt.rb and server/app/lib/utility/oss.rb for their timeout and kill behavior. Done means the listed subprocess paths have consistent timeouts, process-tree cleanup, and the reported fullpath issue is corrected, with tests or existing runner checks passing.