docs: how to use vllm extra body

Use cases, pain points, and background
vllm can accept extra body parameters but it is not obvious how to use this in gym.

To do this, I had to edit vllm_model app:

diff --git a/responses_api_models/vllm_model/app.py b/responses_api_models/vllm_model/app.py
index b9a61f9..2b5984c 100644
--- a/responses_api_models/vllm_model/app.py
+++ b/responses_api_models/vllm_model/app.py
@@ -12,9 +12,10 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+import json
 import re
 from time import time
-from typing import ClassVar, Dict, List, Optional, Tuple, Union
+from typing import Any, ClassVar, Dict, List, Optional, Tuple, Union
 from uuid import uuid4
 
 from aiohttp.client_exceptions import ClientResponseError
@@ -66,6 +67,12 @@ class VLLMModelConfig(BaseResponsesAPIModelConfig):
     uses_reasoning_parser: bool
     replace_developer_role_with_system: bool = False
 
+    chat_template_kwargs: Optional[Dict[str, Any]] = None
+
+    thinking_budget: Optional[int] = None
+    thinking_budget_grace_period: Optional[int] = None
+    end_token_ids: Optional[List[int]] = None
+
     def model_post_init(self, context):
         if isinstance(self.base_url, str):
             self.base_url = [self.base_url]
@@ -198,6 +205,17 @@ class VLLMModel(SimpleResponsesAPIModel):
                 else:
                     raise NotImplementedError
 
+        if self.config.thinking_budget is not None:
+            vllm_xargs = {
+                "thinking_budget": self.config.thinking_budget,
+            }
+            if self.config.thinking_budget_grace_period is not None:
+                vllm_xargs["thinking_budget_grace_period"] = self.config.thinking_budget_grace_period
+            if self.config.end_token_ids is not None:
+                vllm_xargs["end_token_ids"] = json.dumps(self.config.end_token_ids)
+
+            create_params["vllm_xargs"] = vllm_xargs
+
         try:
             chat_completion_dict = await client.create_chat_completion(**create_params)
         except ClientResponseError as e:

next, I made a new vllm_model config that sets these new parameters:

policy_model:
  responses_api_models:
    vllm_model:
      entrypoint: app.py
      base_url: ${policy_base_url}
      api_key: ${policy_api_key}
      model: ${policy_model_name}
      return_token_id_information: true
      uses_reasoning_parser: true

      thinking_budget: 5
      thinking_budget_grace_period: 1
      end_token_ids: [1,2,3]

last, I run using this config and test:

ng_run "+config_paths=[resources_servers/aime25/configs/aime25.yaml,responses_api_models/vllm_model/configs/vllm_model_reasoning_budget.yaml]"


ng_collect_rollouts     +agent_name=aime25_simple_agent +input_jsonl_fpath=resources_servers/aime25/data/train.jsonl +output_jsonl_fpath=results/aime25_rollouts.jsonl  +limit=1

Description:

add a docs section or tutorial explaining how to use this vllm feature

Start with responses_api_models/vllm_model/app.py and the example configuration in responses_api_models/vllm_model/configs/vllm_model_reasoning_budget.yaml. Review the documented ng_run and ng_collect_rollouts commands, then add a tutorial explaining how to configure vLLM extra body parameters such as thinking_budget, thinking_budget_grace_period, and end_token_ids and how to run the example.