lm-sys / lm-sys/FastChat

Arena Error - probabilities contain NaN

Open
#3,294 2 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
39.5k
Forks
4.8k
No merged PRs in 30d

Description

When trying to run arena battle, it errors. The traceback is below

image

2024-04-29 06:11:41 | ERROR | stderr | /workingdir/cen-ai-30032024/aistudent/miniforge3/envs/fastchat/lib/python3.11/site-packages/fastchat/serve/gradio_block_arena_anony.py:304: RuntimeWarning: invalid value encountered in divide
2024-04-29 06:11:41 | ERROR | stderr |   model_weights = model_weights / total_weight
2024-04-29 06:11:41 | ERROR | stderr | Traceback (most recent call last):
2024-04-29 06:11:41 | ERROR | stderr |   File "/workingdir/cen-ai-30032024/aistudent/miniforge3/envs/fastchat/lib/python3.11/site-packages/gradio/queueing.py", line 527, in process_events
2024-04-29 06:11:41 | ERROR | stderr |     response = await route_utils.call_process_api(
2024-04-29 06:11:41 | ERROR | stderr |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-29 06:11:41 | ERROR | stderr |   File "/workingdir/cen-ai-30032024/aistudent/miniforge3/envs/fastchat/lib/python3.11/site-packages/gradio/route_utils.py", line 270, in call_process_api
2024-04-29 06:11:41 | ERROR | stderr |     output = await app.get_blocks().process_api(
2024-04-29 06:11:41 | ERROR | stderr |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-29 06:11:41 | ERROR | stderr |   File "/workingdir/cen-ai-30032024/aistudent/miniforge3/envs/fastchat/lib/python3.11/site-packages/gradio/blocks.py", line 1847, in process_api
2024-04-29 06:11:41 | ERROR | stderr |     result = await self.call_function(
2024-04-29 06:11:41 | ERROR | stderr |              ^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-29 06:11:41 | ERROR | stderr |   File "/workingdir/cen-ai-30032024/aistudent/miniforge3/envs/fastchat/lib/python3.11/site-packages/gradio/blocks.py", line 1433, in call_function
2024-04-29 06:11:41 | ERROR | stderr |     prediction = await anyio.to_thread.run_sync(
2024-04-29 06:11:41 | ERROR | stderr |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-29 06:11:41 | ERROR | stderr |   File "/workingdir/cen-ai-30032024/aistudent/miniforge3/envs/fastchat/lib/python3.11/site-packages/anyio/to_thread.py", line 56, in run_sync
2024-04-29 06:11:41 | ERROR | stderr |     return await get_async_backend().run_sync_in_worker_thread(
2024-04-29 06:11:41 | ERROR | stderr |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-29 06:11:41 | ERROR | stderr |   File "/workingdir/cen-ai-30032024/aistudent/miniforge3/envs/fastchat/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2144, in run_sync_in_worker_thread
2024-04-29 06:11:41 | ERROR | stderr |     return await future
2024-04-29 06:11:41 | ERROR | stderr |            ^^^^^^^^^^^^
2024-04-29 06:11:41 | ERROR | stderr |   File "/workingdir/cen-ai-30032024/aistudent/miniforge3/envs/fastchat/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 851, in run
2024-04-29 06:11:41 | ERROR | stderr |     result = context.run(func, *args)
2024-04-29 06:11:41 | ERROR | stderr |              ^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-29 06:11:41 | ERROR | stderr |   File "/workingdir/cen-ai-30032024/aistudent/miniforge3/envs/fastchat/lib/python3.11/site-packages/gradio/utils.py", line 788, in wrapper
2024-04-29 06:11:41 | ERROR | stderr |     response = f(*args, **kwargs)
2024-04-29 06:11:41 | ERROR | stderr |                ^^^^^^^^^^^^^^^^^^
2024-04-29 06:11:41 | ERROR | stderr |   File "/workingdir/cen-ai-30032024/aistudent/miniforge3/envs/fastchat/lib/python3.11/site-packages/fastchat/serve/gradio_block_arena_anony.py", line 350, in add_text
2024-04-29 06:11:41 | ERROR | stderr |     model_left, model_right = get_battle_pair()
2024-04-29 06:11:41 | ERROR | stderr |                               ^^^^^^^^^^^^^^^^^
2024-04-29 06:11:41 | ERROR | stderr |   File "/workingdir/cen-ai-30032024/aistudent/miniforge3/envs/fastchat/lib/python3.11/site-packages/fastchat/serve/gradio_block_arena_anony.py", line 305, in get_battle_pair
2024-04-29 06:11:41 | ERROR | stderr |     chosen_idx = np.random.choice(len(models), p=model_weights)
2024-04-29 06:11:41 | ERROR | stderr |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-29 06:11:41 | ERROR | stderr |   File "numpy/random/mtrand.pyx", line 971, in numpy.random.mtrand.RandomState.choice
2024-04-29 06:11:41 | ERROR | stderr | ValueError: probabilities contain NaN
2024-04-29 06:11:43 | INFO | gradio_web_server_multi | bot_response_multi (anony). ip: [REDACTED], [REDACTED]
2024-04-29 06:11:43 | ERROR | stderr | Traceback (most recent call last):
2024-04-29 06:11:43 | ERROR | stderr |   File "/workingdir/cen-ai-30032024/aistudent/miniforge3/envs/fastchat/lib/python3.11/site-packages/gradio/queueing.py", line 527, in process_events
2024-04-29 06:11:43 | ERROR | stderr |     response = await route_utils.call_process_api(
2024-04-29 06:11:43 | ERROR | stderr |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-29 06:11:43 | ERROR | stderr |   File "/workingdir/cen-ai-30032024/aistudent/miniforge3/envs/fastchat/lib/python3.11/site-packages/gradio/route_utils.py", line 270, in call_process_api
2024-04-29 06:11:43 | ERROR | stderr |     output = await app.get_blocks().process_api(
2024-04-29 06:11:43 | ERROR | stderr |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-29 06:11:43 | ERROR | stderr |   File "/workingdir/cen-ai-30032024/aistudent/miniforge3/envs/fastchat/lib/python3.11/site-packages/gradio/blocks.py", line 1847, in process_api
2024-04-29 06:11:43 | ERROR | stderr |     result = await self.call_function(
2024-04-29 06:11:43 | ERROR | stderr |              ^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-29 06:11:43 | ERROR | stderr |   File "/workingdir/cen-ai-30032024/aistudent/miniforge3/envs/fastchat/lib/python3.11/site-packages/gradio/blocks.py", line 1445, in call_function
2024-04-29 06:11:43 | ERROR | stderr |     prediction = await utils.async_iteration(iterator)
2024-04-29 06:11:43 | ERROR | stderr |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-29 06:11:43 | ERROR | stderr |   File "/workingdir/cen-ai-30032024/aistudent/miniforge3/envs/fastchat/lib/python3.11/site-packages/gradio/utils.py", line 612, in async_iteration
2024-04-29 06:11:43 | ERROR | stderr |     return await iterator.__anext__()
2024-04-29 06:11:43 | ERROR | stderr |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-29 06:11:43 | ERROR | stderr |   File "/workingdir/cen-ai-30032024/aistudent/miniforge3/envs/fastchat/lib/python3.11/site-packages/gradio/utils.py", line 605, in __anext__
2024-04-29 06:11:43 | ERROR | stderr |     return await anyio.to_thread.run_sync(
2024-04-29 06:11:43 | ERROR | stderr |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-29 06:11:43 | ERROR | stderr |   File "/workingdir/cen-ai-30032024/aistudent/miniforge3/envs/fastchat/lib/python3.11/site-packages/anyio/to_thread.py", line 56, in run_sync
2024-04-29 06:11:43 | ERROR | stderr |     return await get_async_backend().run_sync_in_worker_thread(
2024-04-29 06:11:43 | ERROR | stderr |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-29 06:11:43 | ERROR | stderr |   File "/workingdir/cen-ai-30032024/aistudent/miniforge3/envs/fastchat/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2144, in run_sync_in_worker_thread
2024-04-29 06:11:43 | ERROR | stderr |     return await future
2024-04-29 06:11:43 | ERROR | stderr |            ^^^^^^^^^^^^
2024-04-29 06:11:43 | ERROR | stderr |   File "/workingdir/cen-ai-30032024/aistudent/miniforge3/envs/fastchat/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 851, in run
2024-04-29 06:11:43 | ERROR | stderr |     result = context.run(func, *args)
2024-04-29 06:11:43 | ERROR | stderr |              ^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-29 06:11:43 | ERROR | stderr |   File "/workingdir/cen-ai-30032024/aistudent/miniforge3/envs/fastchat/lib/python3.11/site-packages/gradio/utils.py", line 588, in run_sync_iterator_async
2024-04-29 06:11:43 | ERROR | stderr |     return next(iterator)
2024-04-29 06:11:43 | ERROR | stderr |            ^^^^^^^^^^^^^^
2024-04-29 06:11:43 | ERROR | stderr |   File "/workingdir/cen-ai-30032024/aistudent/miniforge3/envs/fastchat/lib/python3.11/site-packages/gradio/utils.py", line 771, in gen_wrapper
2024-04-29 06:11:43 | ERROR | stderr |     response = next(iterator)
2024-04-29 06:11:43 | ERROR | stderr |                ^^^^^^^^^^^^^^
2024-04-29 06:11:43 | ERROR | stderr |   File "/workingdir/cen-ai-30032024/aistudent/miniforge3/envs/fastchat/lib/python3.11/site-packages/fastchat/serve/gradio_block_arena_anony.py", line 436, in bot_response_multi
2024-04-29 06:11:43 | ERROR | stderr |     state0.to_gradio_chatbot(),
2024-04-29 06:11:43 | ERROR | stderr |     ^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-29 06:11:43 | ERROR | stderr | AttributeError: 'NoneType' object has no attribute 'to_gradio_chatbot'

Arena Side By Side works fine no issues there.

image

Lemme know what are the details you need to debug this. Happy to help!

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in fastchat/serve/gradio_block_arena_anony.py at get_battle_pair and add_text, using the reported traceback to follow model_weights and the subsequent bot_response_multi failure. Reproduce an Arena battle with the reported setup and inspect the values used for probability selection. Done means Arena battle starts and responds without the NaN probability or NoneType errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
ai
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.