Structured extra_records output is discarded for the entire dataset if any single sample fails

Describe the bug

vlmeval/inference.py recognises a structured return shape from model classes — a dict carrying
both prediction and extra_records — via _is_structured_record, and unpacks it into two
columns. The gate is all-or-nothing:

# vlmeval/inference.py
if all(_is_structured_record(data_all[x]) for x in data['index']):
    data['prediction'] = [data_all[x]['prediction'] for x in data['index']]
    data['extra_records'] = [data_all[x]['extra_records'] for x in data['index']]
else:
    data['prediction'] = [str(data_all[x]) for x in data['index']]

A failed sample can never satisfy that predicate, because BaseAPI.generate returns a plain
string once all retries are exhausted:

# vlmeval/api/base.py
return self.fail_msg if answer in ['', None] else answer

So one failed sample anywhere in the dataset makes all(...) False, and every successful
structured record in that run is then passed through str().

This affects a shipped model class

vlmeval/api/arm_thinker.py produces structured records:

extra_records = {"tool_call_count": tool_call_count, "conversation": result}

if rtn:
    ret_code = 0
    return ret_code, rtn, extra_records
else:
    ret_code = 1
    return ret_code, self.fail_msg, extra_records

On success BaseAPI.generate wraps this into {"prediction": ..., "extra_records": ...}. On
failure it returns ret_code = 1, so generate does not accept the result, retries, and finally
falls through to return self.fail_msg — a plain string.

A run therefore only has to contain a single exhausted-retry sample for the whole dataset's
structured output to be destroyed.

Consequences

For a run of N samples in which one sample fails:

• the extra_records column is not created at all;
• the other N−1 predictions become the repr of a dict, e.g.
  "{'prediction': '<answer>', 'extra_records': {'tool_call_count': 3, 'conversation': [...]}}";
• anything downstream that reads the prediction column — including LLM-judge evaluation — scores
  that string rather than the answer.

Nothing signals this. The run exits 0 and writes a normal-looking result file; the only visible
trace is that a column is missing and the remaining scores are wrong. Since transient API failures
(timeouts, transport errors, empty completions) are common at scale, a sufficiently long run is
likely to contain at least one, and one is enough.

Minimal reproduction

The corruption happens in the assembly step and can be shown without any model:

def _is_structured_record(v):
    return isinstance(v, dict) and 'prediction' in v and 'extra_records' in v

data_all = {
    '0': {'prediction': 'A', 'extra_records': {'tool_call_count': 1}},
    '1': {'prediction': 'B', 'extra_records': {'tool_call_count': 2}},
    '2': 'Failed to obtain answer via API.',   # what BaseAPI.generate returns after retries
}

if all(_is_structured_record(v) for v in data_all.values()):
    prediction = [v['prediction'] for v in data_all.values()]
    extra_records = [v['extra_records'] for v in data_all.values()]
else:
    prediction = [str(v) for v in data_all.values()]
    # no extra_records at all

print(prediction[0])
# "{'prediction': 'A', 'extra_records': {'tool_call_count': 1}}"

Drop entry '2' and the same code yields 'A' plus a populated extra_records. The two samples
that succeeded behave completely differently depending on whether an unrelated third sample failed.

Suggested fix

Make the gate per-sample, so a failure only affects its own row:

if any(_is_structured_record(data_all[x]) for x in data['index']):
    data['prediction'] = [
        data_all[x]['prediction'] if _is_structured_record(data_all[x]) else str(data_all[x])
        for x in data['index']
    ]
    data['extra_records'] = [
        data_all[x]['extra_records'] if _is_structured_record(data_all[x]) else {}
        for x in data['index']
    ]
else:
    data['prediction'] = [str(data_all[x]) for x in data['index']]

Guarding with any keeps both homogeneous cases byte-identical to today's behaviour (all
structured, and none structured) and only changes the mixed case. Failed rows keep the fail_msg
string in prediction, so failure-rate accounting is unchanged.

The SPLIT_THINK branch a few lines above has the same all-or-nothing structure and the same issue.

Start in vlmeval/inference.py at the structured-record assembly and the nearby SPLIT_THINK branch, then read vlmeval/api/base.py and vlmeval/api/arm_thinker.py to understand successful and failed returns. Reproduce the mixed structured/plain-string case from the issue. Done means successful rows retain prediction and extra_records, failed rows retain fail_msg, and homogeneous cases keep their current behavior.