gc.collect() causing segmentation fault

**Describe the bug**

DTensorPolicyWorker's `offload_before_refit` call to `gc.collect()` is resulting in a Segmentation fault on A100 pod in a Kubernetes cluster when running the Multiverse Math Hard, Library Judge Math, and Comp Coding GYM environments.

This error does not always occur, it happens about 1/4th of the time during the validation loops.

Reference CI Job where this occurred [here](https://gitlab-master.nvidia.com/aire/microservices/nmp/-/jobs/223616529)

Stack Trace:

```
Running initial validation...
(DTensorPolicyWorker[rank=0] pid=19328) [runner-iekdmqxmu-project-150981-concurrent-15-w1ob93kh:19328:0:19328] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x200211)
(DTensorPolicyWorker[rank=0] pid=19328) ==== backtrace (tid: 19328) ====
(DTensorPolicyWorker[rank=0] pid=19328) 0 /opt/hpcx/nccl_rdma_sharp_plugin/lib/../../ucx/lib/libucs.so.0(ucs_handle_error+0x2e4) [0x7fa1ac08c774]
(DTensorPolicyWorker[rank=0] pid=19328) 1 /opt/hpcx/nccl_rdma_sharp_plugin/lib/../../ucx/lib/libucs.so.0(+0x3796a) [0x7fa1ac08c96a]
(DTensorPolicyWorker[rank=0] pid=19328) 2 /opt/hpcx/nccl_rdma_sharp_plugin/lib/../../ucx/lib/libucs.so.0(+0x37ba8) [0x7fa1ac08cba8]
(DTensorPolicyWorker[rank=0] pid=19328) 3 ray::DTensorPolicyWorker.offload_before_refit() [0x1800a4c]
(DTensorPolicyWorker[rank=0] pid=19328) 4 ray::DTensorPolicyWorker.offload_before_refit() [0x1800829]
(DTensorPolicyWorker[rank=0] pid=19328) 5 ray::DTensorPolicyWorker.offload_before_refit() [0x1a36b0f]
(DTensorPolicyWorker[rank=0] pid=19328) 6 ray::DTensorPolicyWorker.offload_before_refit() [0x188a77a]
(DTensorPolicyWorker[rank=0] pid=19328) 7 ray::DTensorPolicyWorker.offload_before_refit() [0x188a309]
(DTensorPolicyWorker[rank=0] pid=19328) 8 ray::DTensorPolicyWorker.offload_before_refit() [0x1804f67]
(DTensorPolicyWorker[rank=0] pid=19328) 9 ray::DTensorPolicyWorker.offload_before_refit(_PyEval_EvalFrameDefault+0x367f) [0x181617f]
(DTensorPolicyWorker[rank=0] pid=19328) 10 /app/services/customizer/src/nemo-rl/.venv/lib/python3.12/site-packages/ray/_raylet.so(+0x76c6ef) [0x7fa2425e56ef]
(DTensorPolicyWorker[rank=0] pid=19328) 11 /app/services/customizer/src/nemo-rl/.venv/lib/python3.12/site-packages/ray/_raylet.so(+0x79605e) [0x7fa24260f05e]
(DTensorPolicyWorker[rank=0] pid=19328) 12 /app/services/customizer/src/nemo-rl/.venv/lib/python3.12/site-packages/ray/_raylet.so(+0x76c6ef) [0x7fa2425e56ef]
(DTensorPolicyWorker[rank=0] pid=19328) 13 /app/services/customizer/src/nemo-rl/.venv/lib/python3.12/site-packages/ray/_raylet.so(+0x8876c6) [0x7fa2427006c6]
(DTensorPolicyWorker[rank=0] pid=19328) 14 /app/services/customizer/src/nemo-rl/.venv/lib/python3.12/site-packages/ray/_raylet.so(_ZNSt17_Function_handlerIFN3ray6StatusERKNS0_3rpc7AddressENS2_8TaskTypeESsRKNS0_4core11RayFunctionERKSt13unordered_mapISsdSt4hashISsESt8equal_toISsESaISt4pairIKSsdEEERKSt6vectorISt10shared_ptrINS0_9RayObjectEESaISQ_EERKSN_INS2_15ObjectReferenceESaISV_EERSH_S10_PSN_ISG_INS0_8ObjectIDESQ_ESaIS12_EES15_PSN_ISG_IS11_bESaIS16_EERSO_INS0_17LocalMemoryBufferEEPbPSsRKSN_INS0_16ConcurrencyGroupESaIS1F_EESsbbblEPFS1_S5_S6_SsSA_SM_SU_SZ_SsSsS15_S15_S19_S1C_S1D_S1E_S1J_SsbbblEE9_M_invokeERKSt9_Any_dataS5_OS6_OSsSA_SM_SU_SZ_S10_S10_OS15_S1T_OS19_S1C_OS1D_OS1E_S1J_S1S_ObS1X_S1X_Ol+0x169) [0x7fa2425f0369]
(DTensorPolicyWorker[rank=0] pid=19328) 15 /app/services/customizer/src/nemo-rl/.venv/lib/python3.12/site-packages/ray/_raylet.so(_ZN3ray4core10CoreWorker11ExecuteTaskERKNS_17TaskSpecificationESt8optionalISt13unordered_mapISsSt6vectorISt4pairIldESaIS9_EESt4hashISsESt8equal_toISsESaIS8_IKSsSB_EEEEPS7_IS8_INS_8ObjectIDESt10shared_ptrINS_9RayObjectEEESaISP_EESS_PS7_IS8_ISL_bESaIST_EEPN6google8protobuf16RepeatedPtrFieldINS_3rpc20ObjectReferenceCountEEEPbPSs+0xdf5) [0x7fa242819545]
(DTensorPolicyWorker[rank=0] pid=19328) 16 /app/services/customizer/src/nemo-rl/.venv/lib/python3.12/site-packages/ray/_raylet.so(_ZNSt17_Function_handlerIFN3ray6StatusERKNS0_17TaskSpecificationESt8optionalISt13unordered_mapISsSt6vectorISt4pairIldESaIS9_EESt4hashISsESt8equal_toISsESaIS8_IKSsSB_EEEEPS7_IS8_INS0_8ObjectIDESt10shared_ptrINS0_9RayObjectEEESaISP_EESS_PS7_IS8_ISL_bESaIST_EEPN6google8protobuf16RepeatedPtrFieldINS0_3rpc20ObjectReferenceCountEEEPbPSsESt5_BindIFMNS0_4core10CoreWorkerEFS1_S4_SK_SS_SS_SW_S13_S14_S15_EPS19_St12_PlaceholderILi1EES1D_ILi2EES1D_ILi3EES1D_ILi4EES1D_ILi5EES1D_ILi6EES1D_ILi7EES1D_ILi8EEEEE9_M_invokeERKSt9_Any_dataS4_OSK_OSS_S1T_OSW_OS13_OS14_OS15_+0x75) [0x7fa2427a3b45]
(DTensorPolicyWorker[rank=0] pid=19328) 17 /app/services/customizer/src/nemo-rl/.venv/lib/python3.12/site-packages/ray/_raylet.so(+0xa113ce) [0x7fa24288a3ce]
(DTensorPolicyWorker[rank=0] pid=19328) 18 /app/services/customizer/src/nemo-rl/.venv/lib/python3.12/site-packages/ray/_raylet.so(+0xa67912) [0x7fa2428e0912]
(DTensorPolicyWorker[rank=0] pid=19328) 19 /app/services/customizer/src/nemo-rl/.venv/lib/python3.12/site-packages/ray/_raylet.so(_ZN3ray4core20ActorSchedulingQueue31AcceptRequestOrRejectIfCanceledENS_6TaskIDERNS0_14InboundRequestE+0x114) [0x7fa24288f724]
(DTensorPolicyWorker[rank=0] pid=19328) 20 /app/services/customizer/src/nemo-rl/.venv/lib/python3.12/site-packages/ray/_raylet.so(+0xa1c9b3) [0x7fa2428959b3]
(DTensorPolicyWorker[rank=0] pid=19328) 21 /app/services/customizer/src/nemo-rl/.venv/lib/python3.12/site-packages/ray/_raylet.so(_ZN3ray4core20ActorSchedulingQueue3AddEllSt8functionIFvRKNS_17TaskSpecificationES2_IFvNS_6StatusES2_IFvvEES8_EEEES2_IFvS5_RKS6_SA_EESA_S3_+0x5d9) [0x7fa242896239]
(DTensorPolicyWorker[rank=0] pid=19328) 22 /app/services/customizer/src/nemo-rl/.venv/lib/python3.12/site-packages/ray/_raylet.so(_ZN3ray4core12TaskReceiver10HandleTaskENS_3rpc15PushTaskRequestEPNS2_13PushTaskReplyESt8functionIFvNS_6StatusES6_IFvvEES9_EE+0xd03) [0x7fa24288caa3]
(DTensorPolicyWorker[rank=0] pid=19328) 23 /app/services/customizer/src/nemo-rl/.venv/lib/python3.12/site-packages/ray/_raylet.so(+0x984521) [0x7fa2427fd521]
(DTensorPolicyWorker[rank=0] pid=19328) 24 /app/services/customizer/src/nemo-rl/.venv/lib/python3.12/site-packages/ray/_raylet.so(+0xdbcf28) [0x7fa242c35f28]
(DTensorPolicyWorker[rank=0] pid=19328) 25 /app/services/customizer/src/nemo-rl/.venv/lib/python3.12/site-packages/ray/_raylet.so(+0xd67eee) [0x7fa242be0eee]
(DTensorPolicyWorker[rank=0] pid=19328) 26 /app/services/customizer/src/nemo-rl/.venv/lib/python3.12/site-packages/ray/_raylet.so(+0xd68366) [0x7fa242be1366]
(DTensorPolicyWorker[rank=0] pid=19328) 27 /app/services/customizer/src/nemo-rl/.venv/lib/python3.12/site-packages/ray/_raylet.so(+0x1415bbb) [0x7fa24328ebbb]
(DTensorPolicyWorker[rank=0] pid=19328) 28 /app/services/customizer/src/nemo-rl/.venv/lib/python3.12/site-packages/ray/_raylet.so(+0x1417539) [0x7fa243290539]
(DTensorPolicyWorker[rank=0] pid=19328) 29 /app/services/customizer/src/nemo-rl/.venv/lib/python3.12/site-packages/ray/_raylet.so(+0x1417c42) [0x7fa243290c42]
(DTensorPolicyWorker[rank=0] pid=19328) 30 /app/services/customizer/src/nemo-rl/.venv/lib/python3.12/site-packages/ray/_raylet.so(_ZN3ray4core10CoreWorker20RunTaskExecutionLoopEv+0x117) [0x7fa242788b07]
(DTensorPolicyWorker[rank=0] pid=19328) 31 /app/services/customizer/src/nemo-rl/.venv/lib/python3.12/site-packages/ray/_raylet.so(_ZN3ray4core21CoreWorkerProcessImpl26RunWorkerTaskExecutionLoopEv+0x41) [0x7fa242822b81]
(DTensorPolicyWorker[rank=0] pid=19328) 32 /app/services/customizer/src/nemo-rl/.venv/lib/python3.12/site-packages/ray/_raylet.so(_ZN3ray4core17CoreWorkerProcess20RunTaskExecutionLoopEv+0x1d) [0x7fa242822d9d]
(DTensorPolicyWorker[rank=0] pid=19328) 33 /app/services/customizer/src/nemo-rl/.venv/lib/python3.12/site-packages/ray/_raylet.so(+0x770d11) [0x7fa2425e9d11]
(DTensorPolicyWorker[rank=0] pid=19328) 34 ray::DTensorPolicyWorker.offload_before_refit(_PyEval_EvalFrameDefault+0x367f) [0x181617f]
(DTensorPolicyWorker[rank=0] pid=19328) 35 ray::DTensorPolicyWorker.offload_before_refit(PyEval_EvalCode+0xe2) [0x1896a62]
(DTensorPolicyWorker[rank=0] pid=19328) 36 ray::DTensorPolicyWorker.offload_before_refit() [0x18abd42]
(DTensorPolicyWorker[rank=0] pid=19328) 37 ray::DTensorPolicyWorker.offload_before_refit() [0x19b9a93]
(DTensorPolicyWorker[rank=0] pid=19328) 38 ray::DTensorPolicyWorker.offload_before_refit(_PyRun_SimpleFileObject+0x118) [0x19b9818]
(DTensorPolicyWorker[rank=0] pid=19328) 39 ray::DTensorPolicyWorker.offload_before_refit(_PyRun_AnyFileObject+0x42) [0x19b96d0]
(DTensorPolicyWorker[rank=0] pid=19328) 40 ray::DTensorPolicyWorker.offload_before_refit() [0x19b960a]
(DTensorPolicyWorker[rank=0] pid=19328) 41 ray::DTensorPolicyWorker.offload_before_refit() [0x19b951e]
(DTensorPolicyWorker[rank=0] pid=19328) 42 ray::DTensorPolicyWorker.offload_before_refit(Py_RunMain+0x2e5) [0x194a9fb]
(DTensorPolicyWorker[rank=0] pid=19328) 43 ray::DTensorPolicyWorker.offload_before_refit() [0x196593a]
(DTensorPolicyWorker[rank=0] pid=19328) 44 ray::DTensorPolicyWorker.offload_before_refit() [0x196572d]
(DTensorPolicyWorker[rank=0] pid=19328) 45 /usr/lib/x86_64-linux-gnu/libc.so.6(+0x2a1ca) [0x7fa243e431ca]
(DTensorPolicyWorker[rank=0] pid=19328) 46 /usr/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x8b) [0x7fa243e4328b]
(DTensorPolicyWorker[rank=0] pid=19328) 47 ray::DTensorPolicyWorker.offload_before_refit(_start+0x29) [0x19baf29]
(DTensorPolicyWorker[rank=0] pid=19328) =================================
(DTensorPolicyWorker[rank=0] pid=19328) *** SIGSEGV received at time=1761073097 on cpu 202 ***
(DTensorPolicyWorker[rank=0] pid=19328) PC: @ 0x1800a4c (unknown) visit_decref
(DTensorPolicyWorker[rank=0] pid=19328) @ 0x7fa243e5e330 (unknown) (unknown)
(DTensorPolicyWorker[rank=0] pid=19328) [2025-10-21 18:58:17,448 E 19328 19328] logging.cc:496: *** SIGSEGV received at time=1761073097 on cpu 202 ***
(DTensorPolicyWorker[rank=0] pid=19328) [2025-10-21 18:58:17,448 E 19328 19328] logging.cc:496: PC: @ 0x1800a4c (unknown) visit_decref
(DTensorPolicyWorker[rank=0] pid=19328) [2025-10-21 18:58:17,448 E 19328 19328] logging.cc:496: @ 0x7fa243e5e330 (unknown) (unknown)
(DTensorPolicyWorker[rank=0] pid=19328) Fatal Python error: Segmentation fault
(DTensorPolicyWorker[rank=0] pid=19328)
(DTensorPolicyWorker[rank=0] pid=19328) Stack (most recent call first):
(DTensorPolicyWorker[rank=0] pid=19328) File "/app/src/nemo-rl/nemo_rl/models/policy/dtensor_policy_worker.py", line 1865 in offload_before_refit
(DTensorPolicyWorker[rank=0] pid=19328) File "/app/src/nemo-rl/nemo_rl/utils/nsys.py", line 88 in wrapper
(DTensorPolicyWorker[rank=0] pid=19328) File "/app/services/customizer/src/nemo-rl/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116 in decorate_context
(DTensorPolicyWorker[rank=0] pid=19328) File "/app/services/customizer/src/nemo-rl/.venv/lib/python3.12/site-packages/ray/util/tracing/tracing_helper.py", line 463 in _resume_span
(DTensorPolicyWorker[rank=0] pid=19328) File "/app/services/customizer/src/nemo-rl/.venv/lib/python3.12/site-packages/ray/_private/function_manager.py", line 689 in actor_method_executor
(DTensorPolicyWorker[rank=0] pid=19328) File "/app/services/customizer/src/nemo-rl/.venv/lib/python3.12/site-packages/ray/_private/worker.py", line 946 in main_loop
(DTensorPolicyWorker[rank=0] pid=19328) File "/app/services/customizer/src/nemo-rl/.venv/lib/python3.12/site-packages/ray/_private/workers/default_worker.py", line 330 in
(DTensorPolicyWorker[rank=0] pid=19328)
(DTensorPolicyWorker[rank=0] pid=19328) Extension modules: msgpack._cmsgpack, google._upb._message, psutil._psutil_linux, psutil._psutil_posix, setproctitle, yaml._yaml, _brotli, charset_normalizer.md, simplejson._speedups, requests.packages.charset_normalizer.md, requests.packages.chardet.md, uvloop.loop, ray._raylet, numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, regex._regex, markupsafe._speedups, PIL._imaging, sklearn.__check_build._check_build, scipy._lib._ccallback_c, scipy.sparse._sparsetools, _csparsetools, _cyutility, scipy._cyutility, scipy.sparse._csparsetools, scipy.special._ufuncs_cxx, scipy.special._ellip_harm_2, scipy.special._special_ufuncs, scipy.special._gufuncs, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_schur_sqrtm, scipy.linalg._matfuncs_expm, scipy.linalg._linalg_pythran, scipy.linalg.cython_blas, scipy.linalg._decomp_update, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.spatial._ckdtree, scipy._lib.messagestream, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._hausdorff, scipy.spatial._distance_wrap, scipy.spatial.transform._rotation, scipy.spatial.transform._rigid_transform, scipy.optimize._group_columns, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._slsqplib, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy._lib._uarray._uarray, scipy.linalg._decomp_interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.optimize._direct, scipy.integrate._odepack, scipy.integrate._quadpack, scipy.integrate._vode, scipy.integrate._dop, scipy.integrate._lsoda, scipy.interpolate._fitpack, scipy.interpolate._dfitpack, scipy.interpolate._dierckx, scipy.interpolate._ppoly, scipy.interpolate._interpnd, scipy.interpolate._rbfinterp_pythran, scipy.interpolate._rgi_cython, scipy.special.cython_special, scipy.stats._stats, scipy.stats._biasedurn, scipy.stats._stats_pythran, scipy.stats._levy_stable.levyst, scipy.stats._ansari_swilk_statistics, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, scipy.stats._sobol, scipy.stats._qmc_cy, scipy.stats._rcont.rcont, scipy.stats._qmvnt_cy, scipy.ndimage._nd_image, scipy.ndimage._rank_filter_1d, _ni_label, scipy.ndimage._ni_label, pyarrow.lib, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pyarrow._compute, pandas._libs.ops, pandas._libs.hashing, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.internals, pandas._libs.indexing, pandas._libs.index, pandas._libs.writers, pandas._libs.join, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing, sklearn._cyutility, sklearn.utils._isfinite, sklearn.utils.sparsefuncs_fast, sklearn.utils.murmurhash, sklearn.utils._openmp_helpers, sklearn.metrics.cluster._expected_mutual_info_fast, sklearn.preprocessing._csr_polynomial_expansion, sklearn.preprocessing._target_encoder_fast, sklearn.metrics._dist_metrics, sklearn.metrics._pairwise_distances_reduction._datasets_pair, sklearn.utils._cython_blas, sklearn.metrics._pairwise_distances_reduction._base, sklearn.metrics._pairwise_distances_reduction._middle_term_computer, sklearn.utils._heap, sklearn.utils._sorting, sklearn.metrics._pairwise_distances_reduction._argkmin, sklearn.metrics._pairwise_distances_reduction._argkmin_classmode, sklearn.utils._vector_sentinel, sklearn.metrics._pairwise_distances_reduction._radius_neighbors, sklearn.metrics._pairwise_distances_reduction._radius_neighbors_classmode, sklearn.metrics._pairwise_fast, PIL._imagingft, _cffi_backend, pyarrow._json (total: 192)
```

Config passed to `run_grpo_penguin.py`

```
grpo:
max_num_epochs: 1
max_num_steps: 375
val_period: 186
seed: 42
val_batch_size: null
val_at_start: true
max_val_samples: null
num_generations_per_prompt: 4
num_prompts_per_step: 2
normalize_rewards: true
use_leave_one_out_baseline: true
overlong_filtering: false
max_rollout_turns: 1
loss_fn:
reference_policy_kl_penalty: 0.005
ratio_clip_min: 0.21
ratio_clip_max: 0.21
ratio_clip_c: null
use_on_policy_kl_approximation: false
use_importance_sampling_correction: false
token_level_loss: true
checkpointing:
enabled: true
checkpoint_dir: /app/workspace/cust-8SGiiY14ZUesbGCxDiqemd/workspace/checkpoints
metric_name: val_reward
higher_is_better: true
keep_top_k: 1
save_period: 187
checkpoint_must_save_by: null
policy:
model_name: /app/workspace/cust-8SGiiY14ZUesbGCxDiqemd/workspace/hf
tokenizer:
name: /app/workspace/cust-8SGiiY14ZUesbGCxDiqemd/workspace/hf
chat_template: "{{- bos_token }}\n{%- if not date_string is defined %}\n {%-\
\ if strftime_now is defined %}\n {%- set date_string = strftime_now(\"\
%d %b %Y\") %}\n {%- else %}\n {%- set date_string = \"26 Jul 2024\"\
\ %}\n {%- endif %}\n{%- endif %}\n{%- set loop_messages = messages %}\n\
{%- if tools is not none and tool_choice is not none %}\n {{- '<|start_header_id|>system<|end_header_id|>\\\
n\\n' }}\n {{- \"Environment: ipython\\n\\n\" }}\n {{- \"Cutting Knowledge\
\ Date: December 2023\\n\" }}\n {{- \"Today Date: \" + date_string + \"\\\
n\\n\" }}\n {{- \"You are a helpful assistant.\\n\" }}\n {{- '<|eot_id|>'\
\ }}\n {{- '<|start_header_id|>user<|end_header_id|>\\n\\n' }}\n {{- 'You\
\ have access to the following functions to supplement your existing knowledge:\\\
n\\n' }}\n {%- for t in tools %}\n {%- set tname = t.function.name\
\ %}\n {%- set tdesc = t.function.description %}\n {%- set tparams\
\ = t.function.parameters | tojson %}\n {{- \"Use the function '\" +\
\ tname + \"' to '\" + tdesc + \"':\\n\" }}\n {{- '{\"name\": \"' + tname\
\ + '\", \"description\": \"' + tdesc + '\", \"parameters\": ' + tparams + '}\\\
n\\n' }}\n {%- endfor %}\n {{- 'Think very carefully before calling functions.\\\
n' }}\n {{- 'Only call them if they are relevant to the prompt.\\n' }}\n\
\ {{- 'If you choose to call a function ONLY reply in the following format\
\ with no natural language surrounding it:\\n\\n' }}\n {{- '{\"\
example_name\": \"example_value\"}\\n\\n' }}\n {{- 'Reminder:\\\
n' }}\n {{- '- Function calls MUST follow the specified format, start with\
\ \\n' }}\n {{- '- Required parameters\
\ MUST be specified\\n' }}\n {{- '- Only call one function at a time\\n'\
\ }}\n {{- '- Put the entire function call reply on one line\\n' }}\n \
\ {{- '- Do not call functions if they are not relevant to the prompt' }}\n\
\ {{- '<|eot_id|>' }}\n{%- endif %}\n{%- for message in loop_messages %}\n\
\ {%- if message['role'] in ['ipython', 'tool'] %}\n {{- \"<|start_header_id|>ipython<|end_header_id|>\\\
n\\n\" }}\n {{- \"[stdout]\" + message['content'] | trim + \"[/stdout]\\\
n<|eot_id|>\" }}\n {%- elif message['role'] == 'assistant'%}\n {{-\
\ '<|start_header_id|>assistant<|end_header_id|>\\n\\n' }}\n {%- if message.get('tool_calls')\
\ is not none %}\n {%- set tool_call = message['tool_calls'][0] %}\n\
\ {%- generation %}\n {{- '<|python_tag|>' + tool_call.function.arguments | tojson +\
\ '\\n<|eot_id|>' }}\n {%- endgeneration %}\n {%-\
\ else %}\n {%- generation %}\n {{- message['content']\
\ | trim + '<|eot_id|>' }}\n {%- endgeneration %}\n {%- endif\
\ %}\n {%- else %}\n {{- '<|start_header_id|>' + message['role'] +\
\ '<|end_header_id|>\\n\\n' }}\n {{- message['content'] | trim + '<|eot_id|>'\
\ }}\n {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n \
\ {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' }}\n{%- endif %}\n"
train_global_batch_size: ${mul:${grpo.num_prompts_per_step}, ${grpo.num_generations_per_prompt}}
train_micro_batch_size: 1
generation_batch_size: 16
logprob_batch_size: 1
max_total_sequence_length: 4096
precision: bfloat16
logprob_chunk_size: 1024
dtensor_cfg:
enabled: true
_v2: false
cpu_offload: false
sequence_parallel: false
activation_checkpointing: false
tensor_parallel_size: 1
context_parallel_size: 1
custom_parallel_plan: null
clear_cache_every_n_steps: null
megatron_cfg:
enabled: false
dynamic_batching:
enabled: false
sequence_packing:
enabled: false
make_sequence_length_divisible_by: ${policy.dtensor_cfg.tensor_parallel_size}
max_grad_norm: 0.99
generation:
backend: vllm
max_new_tokens: ${policy.max_total_sequence_length}
temperature: 0.9
top_p: 1.0
top_k: null
stop_token_ids: None
stop_strings: None
vllm_cfg:
async_engine: true
precision: ${policy.precision}
tensor_parallel_size: 1
pipeline_parallel_size: 1
enable_expert_parallel: false
expert_parallel_size: 1
gpu_memory_utilization: 0.8
max_model_len: ${policy.max_total_sequence_length}
enforce_eager: false
use_deep_gemm: false
num_last_layers_in_bf16: 0
num_first_layers_in_bf16: 0
expose_http_server: true
http_server_serving_chat_kwargs:
enable_auto_tools: true
tool_parser: llama3_json
vllm_kwargs:
compilation_config:
use_inductor: false
colocated:
enabled: true
resources:
gpus_per_node: null
num_nodes: null
optimizer:
kwargs:
lr: 5.0e-06
weight_decay: 0.01
betas:
- 0.9
- 0.99
eps: 1.0e-05
foreach: false
fused: false
name: torch.optim.AdamW
scheduler:
- name: torch.optim.lr_scheduler.LinearLR
kwargs:
start_factor: 1.0e-05
end_factor: 1.0
total_iters: 5
- name: torch.optim.lr_scheduler.CosineAnnealingLR
kwargs:
T_max: 100000000000
eta_min: 0.0
- milestones:
- 5
data:
max_input_seq_length: ${policy.max_total_sequence_length}
train_jsonl_fpath: /builds/aire/microservices/nmp/data/cust-8SGiiY14ZUesbGCxDiqemd/merged/training.jsonl
validation_jsonl_fpath: /builds/aire/microservices/nmp/data/cust-8SGiiY14ZUesbGCxDiqemd/merged/validation.jsonl
shuffle: false
env:
should_use_penguin: true
penguin:
config_paths:
- responses_api_models/vllm_model/configs/vllm_model_for_training.yaml
- resources_servers/library_judge_math/configs/library_judge_math.yaml
library_judge_math:
resources_servers:
library_judge_math:
judge_model_server:
type: responses_api_models
name: policy_model
judge_responses_create_params:
input: []
temperature: 1.0
top_p: 1.0
should_use_judge: false
domain: math
logger:
log_dir: /app/workspace/cust-8SGiiY14ZUesbGCxDiqemd/workspace/logs
num_val_samples_to_print: 0
wandb_enabled: false
mlflow_enabled: false
tensorboard_enabled: false
monitor_gpus: false
swanlab_enabled: false
cluster:
gpus_per_node: 1
num_nodes: 1
```

`/app/workspace/cust-8SGiiY14ZUesbGCxDiqemd/workspace/hf` is a Huggingface Llama 3.2 - 1b - Instruct model