How to evaluate MathVista with GPT-like closed-source api?

Hi, I tried the following command and got the response successfully.

>>>messages = [{'role': 'user', 'content': 'what is your name?'}]
>>> openai_client = openai.OpenAI(api_key='xxx',base_url="https:/xxx.com/api/openai/v1")
>>> response = openai_client.chat.completions.create(model='gpt-3.5-turbo', messages=messages)
>>> print(response)
ChatCompletion(id=None, choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='I am an AI digital assistant and do not have a personal name. You can refer to me as Assistant.', refusal=None, role='assistant', annotations=None, audio=None, function_call=None, tool_calls=None))], created=None, model=None, object=None, service_tier=None, system_fingerprint=None, usage=CompletionUsage(completion_tokens=22, prompt_tokens=10, total_tokens=32, completion_tokens_details=None, prompt_tokens_details=None, cache_creation_input_tokens=None, cache_read_input_tokens=None))

However, when I set key and api-base as environment variables,

export OPENAI_API_KEY=xxx
python run.py --data MathVista_MINI --model Qwen2.5-VL-7B-Instruct --verbose

and update gpt.py

APIBASES = {
    # 'OFFICIAL': 'https://api.openai.com/v1/chat/completions',
    'OFFICIAL': 'https://xxx/api/openai/v1',
}

it doesn't work.

The error message:

Infer Qwen2.5-VL-7B-Instruct/MathVista_MINI, Rank 0/1: 100%|█████████▉| 999/1000 [1:51:41<00:06, 6.46s/it]
Infer Qwen2.5-VL-7B-Instruct/MathVista_MINI, Rank 0/1: 100%|██████████| 1000/1000 [1:51:42<00:00, 4.89s/it]
Infer Qwen2.5-VL-7B-Instruct/MathVista_MINI, Rank 0/1: 100%|██████████| 1000/1000 [1:51:42<00:00, 6.70s/it]
[2025-05-23 02:35:33] INFO - RUN - run.py: main - 398: {'nproc': 4, 'verbose': True, 'retry': 3, 'model': 'gpt-4o-mini'}
[2025-05-23 02:35:33] ERROR - misc.py: load_env - 219: Did not detect the .env file at /checkpoint/binary/train_package/.env, failed to load.
[2025-05-23 02:35:33] INFO - ChatAPI - gpt.py: init - 147: Using API Base: https://xxx.com/api/openai/v1; API Key: sk-xxx
[2025-05-23 02:35:34] ERROR - ChatAPI - gpt.py: generate_inner - 222: <class 'KeyError'>: 'choices'
[2025-05-23 02:35:34] ERROR - ChatAPI - gpt.py: generate_inner - 223: {"timestamp":"2025-05-22T18:35:34.268+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
[2025-05-23 02:35:34] INFO - ChatAPI - base.py: generate - 257: RetCode: 404
Answer: Failed to obtain answer via API.
Log: {"timestamp":"2025-05-22T18:35:34.268+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
[2025-05-23 02:35:42] ERROR - ChatAPI - gpt.py: generate_inner - 222: <class 'KeyError'>: 'choices'
[2025-05-23 02:35:42] ERROR - ChatAPI - gpt.py: generate_inner - 223: {"timestamp":"2025-05-22T18:35:42.988+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
[2025-05-23 02:35:42] INFO - ChatAPI - base.py: generate - 257: RetCode: 404
Answer: Failed to obtain answer via API.
Log: {"timestamp":"2025-05-22T18:35:42.988+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
[2025-05-23 02:35:46] ERROR - ChatAPI - gpt.py: generate_inner - 222: <class 'KeyError'>: 'choices'
[2025-05-23 02:35:46] ERROR - ChatAPI - gpt.py: generate_inner - 223: {"timestamp":"2025-05-22T18:35:46.323+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
[2025-05-23 02:35:46] INFO - ChatAPI - base.py: generate - 257: RetCode: 404
Answer: Failed to obtain answer via API.
Log: {"timestamp":"2025-05-22T18:35:46.323+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
[2025-05-23 02:35:47] ERROR - ChatAPI - gpt.py: generate_inner - 222: <class 'KeyError'>: 'choices'
[2025-05-23 02:35:47] ERROR - ChatAPI - gpt.py: generate_inner - 223: {"timestamp":"2025-05-22T18:35:47.390+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
[2025-05-23 02:35:47] INFO - ChatAPI - base.py: generate - 257: RetCode: 404
Answer: Failed to obtain answer via API.
Log: {"timestamp":"2025-05-22T18:35:47.390+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
[2025-05-23 02:35:51] ERROR - ChatAPI - gpt.py: generate_inner - 222: <class 'KeyError'>: 'choices'
[2025-05-23 02:35:51] ERROR - ChatAPI - gpt.py: generate_inner - 223: {"timestamp":"2025-05-22T18:35:51.178+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
[2025-05-23 02:35:51] INFO - ChatAPI - base.py: generate - 257: RetCode: 404
Answer: Failed to obtain answer via API.
Log: {"timestamp":"2025-05-22T18:35:51.178+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
[2025-05-23 02:35:52] ERROR - ChatAPI - gpt.py: generate_inner - 222: <class 'KeyError'>: 'choices'
[2025-05-23 02:35:52] ERROR - ChatAPI - gpt.py: generate_inner - 223: {"timestamp":"2025-05-22T18:35:52.337+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
[2025-05-23 02:35:52] INFO - ChatAPI - base.py: generate - 257: RetCode: 404
Answer: Failed to obtain answer via API.
Log: {"timestamp":"2025-05-22T18:35:52.337+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
[2025-05-23 02:36:01] ERROR - ChatAPI - gpt.py: generate_inner - 222: <class 'KeyError'>: 'choices'
[2025-05-23 02:36:01] ERROR - ChatAPI - gpt.py: generate_inner - 223: {"timestamp":"2025-05-22T18:36:01.070+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
[2025-05-23 02:36:01] INFO - ChatAPI - base.py: generate - 257: RetCode: 404
Answer: Failed to obtain answer via API.
Log: {"timestamp":"2025-05-22T18:36:01.070+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
[2025-05-23 02:36:01] ERROR - ChatAPI - gpt.py: generate_inner - 222: <class 'KeyError'>: 'choices'
[2025-05-23 02:36:01] ERROR - ChatAPI - gpt.py: generate_inner - 223: {"timestamp":"2025-05-22T18:36:01.228+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
[2025-05-23 02:36:01] INFO - ChatAPI - base.py: generate - 257: RetCode: 404
Answer: Failed to obtain answer via API.
Log: {"timestamp":"2025-05-22T18:36:01.228+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
[2025-05-23 02:36:08] ERROR - ChatAPI - gpt.py: generate_inner - 222: <class 'KeyError'>: 'choices'
[2025-05-23 02:36:08] ERROR - ChatAPI - gpt.py: generate_inner - 223: {"timestamp":"2025-05-22T18:36:08.129+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
[2025-05-23 02:36:08] INFO - ChatAPI - base.py: generate - 257: RetCode: 404
Answer: Failed to obtain answer via API.
Log: {"timestamp":"2025-05-22T18:36:08.129+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
[2025-05-23 02:36:15] ERROR - ChatAPI - gpt.py: generate_inner - 222: <class 'KeyError'>: 'choices'
[2025-05-23 02:36:15] ERROR - ChatAPI - gpt.py: generate_inner - 223: {"timestamp":"2025-05-22T18:36:15.114+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
[2025-05-23 02:36:15] INFO - ChatAPI - base.py: generate - 257: RetCode: 404
Answer: Failed to obtain answer via API.
Log: {"timestamp":"2025-05-22T18:36:15.114+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
[2025-05-23 02:36:15] ERROR - ChatAPI - gpt.py: generate_inner - 222: <class 'KeyError'>: 'choices'
[2025-05-23 02:36:15] ERROR - ChatAPI - gpt.py: generate_inner - 223: {"timestamp":"2025-05-22T18:36:15.223+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
[2025-05-23 02:36:15] INFO - ChatAPI - base.py: generate - 257: RetCode: 404
Answer: Failed to obtain answer via API.
Log: {"timestamp":"2025-05-22T18:36:15.223+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
[2025-05-23 02:36:23] ERROR - ChatAPI - gpt.py: generate_inner - 222: <class 'KeyError'>: 'choices'
[2025-05-23 02:36:23] ERROR - ChatAPI - gpt.py: generate_inner - 223: {"timestamp":"2025-05-22T18:36:23.494+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
[2025-05-23 02:36:23] INFO - ChatAPI - base.py: generate - 257: RetCode: 404
Answer: Failed to obtain answer via API.
Log: {"timestamp":"2025-05-22T18:36:23.494+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
[2025-05-23 02:36:25] ERROR - ChatAPI - gpt.py: generate_inner - 222: <class 'KeyError'>: 'choices'
[2025-05-23 02:36:25] ERROR - ChatAPI - gpt.py: generate_inner - 223: {"timestamp":"2025-05-22T18:36:25.175+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
[2025-05-23 02:36:25] INFO - ChatAPI - base.py: generate - 257: RetCode: 404
Answer: Failed to obtain answer via API.
Log: {"timestamp":"2025-05-22T18:36:25.175+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
[2025-05-23 02:36:29] ERROR - ChatAPI - gpt.py: generate_inner - 222: <class 'KeyError'>: 'choices'
[2025-05-23 02:36:29] ERROR - ChatAPI - gpt.py: generate_inner - 223: {"timestamp":"2025-05-22T18:36:29.783+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
[2025-05-23 02:36:29] INFO - ChatAPI - base.py: generate - 257: RetCode: 404
Answer: Failed to obtain answer via API.
Log: {"timestamp":"2025-05-22T18:36:29.783+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
[2025-05-23 02:36:37] ERROR - ChatAPI - gpt.py: generate_inner - 222: <class 'KeyError'>: 'choices'
[2025-05-23 02:36:37] ERROR - ChatAPI - gpt.py: generate_inner - 223: {"timestamp":"2025-05-22T18:36:37.163+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
[2025-05-23 02:36:37] INFO - ChatAPI - base.py: generate - 257: RetCode: 404
Answer: Failed to obtain answer via API.
Log: {"timestamp":"2025-05-22T18:36:37.163+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
[2025-05-23 02:36:41] ERROR - RUN - run.py: main - 478: Model Qwen2.5-VL-7B-Instruct x Dataset MathVista_MINI combination failed: MathVista evaluation requires a working OPENAI API

To debug the OpenAI API, you can try the following scripts in python:

from vlmeval.api import OpenAIWrapper
model = OpenAIWrapper('gpt-4o', verbose=True)
msgs = [dict(type='text', value='Hello!')]
code, answer, resp = model.generate_inner(msgs)
print(code, answer, resp)

You cam see the specific error if the API call fails.
, skipping this combination.
Traceback (most recent call last):
File "/checkpoint/binary/train_package/run.py", line 449, in main
eval_results = dataset.evaluate(result_file, **judge_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/checkpoint/binary/train_package/vlmeval/dataset/image_vqa.py", line 229, in evaluate
assert model.working(), ('MathVista evaluation requires a working OPENAI API\n' + DEBUG_MESSAGE)
^^^^^^^^^^^^^^^
AssertionError: MathVista evaluation requires a working OPENAI API

To debug the OpenAI API, you can try the following scripts in python:

from vlmeval.api import OpenAIWrapper
model = OpenAIWrapper('gpt-4o', verbose=True)
msgs = [dict(type='text', value='Hello!')]
code, answer, resp = model.generate_inner(msgs)
print(code, answer, resp)

You cam see the specific error if the API call fails.

[2025-05-23 02:36:44,577] [WARNING] [AiScheduler#26] [heart_beat_client_thread.py:120] HeartbeatClient: Failed to send finial message on stop status: internal_exception, detail: Internal program exception, detail: KeyError('RANK')

Start with vlmeval/api.OpenAIWrapper and gpt.py, then inspect the debug call shown in the issue and the MathVista check in vlmeval/dataset/image_vqa.py. Reproduce the request using run.py or the standalone OpenAIWrapper snippet, and consider the work complete when the configured API returns the expected response and MathVista evaluation proceeds.