code_gen extract_code picks last code block instead of longest — evaluates test snippets instead of solutions

Bug Description

extract_code in resources_servers/code_gen/lcb_integration/extraction_utils.py picks the code between the last two triple-backtick markers. When a model generates its solution in one code block and then appends a short test/demo snippet in a second code block, the extraction evaluates the test snippet instead of the actual solution.

This corrupts both the reward (correct solutions get reward 0.0) and the feedback text (feedback describes errors in the test snippet, not the solution). This affects any RL algorithm using the code_gen environment (GRPO, etc.).

Evidence

Analysis of 100 RL training samples (Qwen3-8B on LiveCodeBench v6):

In all 18 cases, the longest block is the real solution (contains class Solution, def, imports) and the last block is a short test snippet. All 18 got "Runtime Error" feedback because the test snippet isn't a valid solution.

Reproduction

from lcb_integration.extraction_utils import extract_code
from lcb_integration.lm_styles import LMStyle

# Simulate a typical model response with solution + test snippet
model_output = (
    "Here is my solution:\n\n"
    "```python\n"
    "from typing import List\n\n"
    "class Solution:\n"
    "    def maxSum(self, nums: List[int], k: int, m: int) -> int:\n"
    "        n = len(nums)\n"
    "        prefix = [0] * (n + 1)\n"
    "        for i in range(n):\n"
    "            prefix[i + 1] = prefix[i] + nums[i]\n"
    "        return 0  # placeholder\n"
    "```\n\n"
    "Let me test it:\n\n"
    "```python\n"
    "sol = Solution()\n"
    'print(sol.maxSum([1, 2, -1, 3, 3, 4], 2, 2))  # Output: 13\n'
    "```\n"
)

code = extract_code(model_output, LMStyle.OpenAIChat)
print(repr(code))
# ACTUAL:   'sol = Solution()\nprint(sol.maxSum([1, 2, -1, 3, 3, 4], 2, 2))  # Output: 13'
# EXPECTED: the full Solution class (the longest block)

Root Cause

Line 31 of extraction_utils.py:

return "\n".join(outputlines[indexlines[-2] + 1 : indexlines[-1]])

This always picks the content between the last two ``` markers, regardless of block length or content.

Suggested Fix

Pick the longest code block instead of the last one:

# Current (picks last block):
return "\n".join(outputlines[indexlines[-2] + 1 : indexlines[-1]])

# Proposed (picks longest block):
blocks = []
for j in range(0, len(indexlines) - 1, 2):
    block = "\n".join(outputlines[indexlines[j] + 1 : indexlines[j + 1]])
    blocks.append(block)
if not blocks:
    return ""
return max(blocks, key=len)

Impact

• 18% of samples in our training runs had corrupted rewards and feedback due to this bug
• Affects all RL algorithms using the code_gen resource server (GRPO, etc.)
• Correct solutions may receive reward 0.0 because a test snippet is evaluated instead
• Feedback text describes errors in the test snippet, misleading the model during training

Start in resources_servers/code_gen/lcb_integration/extraction_utils.py at extract_code and reproduce the supplied multi-block model output. Verify that extraction selects the longest fenced block, preserves the single-block behavior, and returns the solution rather than the trailing test snippet.