NatLabRockies / NatLabRockies/openstudio-server-helm
[Feature Request] Add optional Kubernetes Indexed Job template for external_batch simulation execution
@anchapin is already working on this.
Since Aug 16, 2026.
- Dominant language
- Go Template
- Stars
- 12
- Forks
- 24
- PR merge metrics
- No merged PRs in 30d
Description
Description
The openstudio-server project introduced the external_batch execution mode, which packages analysis datapoints into self-contained chunk archives (run_chunk.rb) to eliminate worker starvation and end-of-analysis compute valleys.
This issue proposes adding a new optional Kubernetes Job template (templates/job-external-batch.yaml) and corresponding configuration in values.yaml to allow running external batch chunks natively on Kubernetes alongside or in place of the standard Resque worker deployment, reusing the chart's existing NFS storage.
Motivation & Benefits
- Compute Valley Mitigation: Shifts job scheduling to Kubernetes, allowing analysis batches to overlap execution without bottlenecking the
web-backgroundResque processes. - Zero Extra Infrastructure: Reuses the existing Kubernetes cluster and Helm release without needing an external batch cluster (such as AWS Batch, SLURM, or Nomad).
- Dual-Mode Coexistence: Users can continue running traditional Resque worker deployments for iterative/standard analyses (
batch_run,ga,pso) while executingexternal_batch_runworkloads on-demand via indexed Kubernetes Jobs. - Storage Reuse: Job pods mount the existing shared NFS Persistent Volume Claim (
ReadWriteMany), avoiding duplicate data transfer.
Proposed Changes
1. values.yaml additions
Add a dedicated configuration section for external_batch:
external_batch:
enabled: false
analysis_id: ""
completions: 1 # Total number of chunks generated by the packager
parallelism: 1 # Concurrency / number of chunk pods running simultaneously
backoff_limit: 3
active_deadline_seconds: 86400
ttl_seconds_after_finished: 3600
container:
image: nrel/openstudio-server:latest
imagePullPolicy: IfNotPresent
storage:
mountPath: /shared-data
# Reuses existing chart NFS PVC if left blank
existingClaim: ""
resources:
requests:
cpu: "2000m"
memory: "4Gi"
limits:
cpu: "4000m"
memory: "8Gi"
**2. New Template: templates/job-external-batch.yaml**
Leverage Kubernetes completionMode: Indexed so the environment variable JOB_COMPLETION_INDEX dynamically matches the chunk ID:
{{- if .Values.external_batch.enabled }}
apiVersion: batch/v1
kind: Job
metadata:
name: {{ include "openstudio-server.fullname" . }}-batch-{{ .Values.external_batch.analysis_id | default "manual" }}
labels:
app.kubernetes.io/name: {{ include "openstudio-server.name" . }}-external-batch
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/component: simulation-job
spec:
completions: {{ .Values.external_batch.completions }}
parallelism: {{ .Values.external_batch.parallelism }}
completionMode: Indexed
backoffLimit: {{ .Values.external_batch.backoff_limit }}
activeDeadlineSeconds: {{ .Values.external_batch.active_deadline_seconds }}
ttlSecondsAfterFinished: {{ .Values.external_batch.ttl_seconds_after_finished }}
template:
metadata:
labels:
app.kubernetes.io/name: {{ include "openstudio-server.name" . }}-external-batch
app.kubernetes.io/instance: {{ .Release.Name }}
spec:
restartPolicy: OnFailure
containers:
- name: chunk-runner
image: {{ .Values.external_batch.container.image | default .Values.worker.container.image }}
imagePullPolicy: {{ .Values.external_batch.container.imagePullPolicy }}
command: ["/bin/bash", "-c"]
args:
- |
ruby /openstudio-server/external_batch/runner/run_chunk.rb \
--analysis-id {{ .Values.external_batch.analysis_id }} \
--chunk-id ${JOB_COMPLETION_INDEX} \
--storage-dir {{ .Values.external_batch.storage.mountPath }}
resources:
{{- toYaml .Values.external_batch.resources | nindent 12 }}
volumeMounts:
- name: shared-nfs
mountPath: {{ .Values.external_batch.storage.mountPath }}
volumes:
- name: shared-nfs
persistentVolumeClaim:
claimName: {{ .Values.external_batch.storage.existingClaim | default (printf "%s-nfs" (include "openstudio-server.fullname" .)) }}
{{- end }}
Usage Example (Ad-Hoc Execution)
Users can template and dispatch batch jobs on demand for a packaged analysis without modifying the running release:
helm template openstudio-server ./openstudio-server-helm \
-s templates/job-external-batch.yaml \
--set external_batch.enabled=true \
--set external_batch.analysis_id="<ANALYSIS_UUID>" \
--set external_batch.completions=16 \
--set external_batch.parallelism=8 | kubectl apply -f -
Acceptance Criteria
-
external_batchvalues block defined invalues.yaml(defaultenabled: false). -
templates/job-external-batch.yamladded usingcompletionMode: Indexed. - Job pods successfully mount the existing NFS PVC and access analysis chunk packages.
- Unit tests added verifying
helm templaterenders the Job only when enabled. - Documentation added to
README.mdexplaining dual-mode operation and external batch dispatch.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.