sglang worker raise exception in benchmark

Nobody has claimed this yet.

I use this script to benchmark the sglang worker

Script

`--dataset` is `ShareGPT_V3_unfiltered_cleaned_split.json`. Modified from vllm

Command used
```
python3 -m fastchat.serve.sglang_worker --model-path Llama-2-7b-chat-hf/ --host 0.0.0.0 --port 30000 --worker-address http://localhost:30000 --controller http://localhost:21000 --num-gpus 8

python3 benchmark.py --host localhost --port 30000 --count 1000 --dataset ./ShareGPT_V3_unfiltered_cleaned_split.json
```

```python
import argparse
import asyncio
import aiohttp
import json
import random
import os

os.environ['http_proxy'] = ''
os.environ['https_proxy'] = ''
os.environ['all_proxy'] = ''

async def post_request(url, data):
timeout = aiohttp.ClientTimeout(total=3 * 3600)
headers = {"User-Agent": "Benchmark Client"}
async with aiohttp.ClientSession(timeout=timeout) as session:
async with session.post(url, headers=headers, json=data) as response:
chunks = []
async for chunk, _ in response.content.iter_chunks():
chunks.append(chunk)
output = b"".join(chunks).decode("utf-8")
output = json.loads(output)

return output

def sample_request(dataset_path, count):
with open(dataset_path) as f:
dataset = json.load(f)

random.seed(123456)
dataset = random.sample(dataset, count * 3)
prompt = [
data["conversations"][0]["value"] for data in dataset
if len(data["conversations"]) >= 2 and len(data["conversations"][0]["value"]) > 10 and data["conversations"][0]['from'] == 'human'
]
prompt = random.sample(prompt, count)
return prompt

async def send_request_to_worker(
prompt: str,
api_url: str,
):
data = {
'prompt': prompt,
'temperature': 0.0,
'max_new_tokens': 512,
}

await post_request(api_url, data)

async def benchmark(
host,
port,
count,
dataset_path,
) -> None:
tasks = []

send_request = send_request_to_worker
api_url = f"http://{host}:{port}/worker_generate"

request_prompt = sample_request(dataset_path, count)

for prompt in request_prompt:
task = asyncio.create_task(send_request(prompt, api_url))
tasks.append(task)
await asyncio.sleep(0.01)
await asyncio.gather(*tasks)

if __name__ == "__main__":
parser = argparse.ArgumentParser(
description="Benchmark the online serving throughput.")
parser.add_argument("--host", type=str, default="localhost")
parser.add_argument("--port", type=int, default=8000)
parser.add_argument("--count", type=int, default=1000)
parser.add_argument("--dataset", type=str, required=True,
help="Path to the dataset.")
args = parser.parse_args()

asyncio.run(benchmark(args.host, args.port, args.count, args.dataset))
```

But it raise the exception `UnboundLocalError: local variable 'x' referenced before assignment` in https://github.com/lm-sys/FastChat/blob/9924687b67d62032641640bd245d682c4d2f025e/fastchat/serve/sglang_worker.py#L164 It seems that the for loop exit directly instead yielding anything. I didn't try to find out whether it is the problem related to sglang.

Another issue comes from the sglang itself, but as the maintainers are overlapped across these two repos, so I post it here.

In https://github.com/sgl-project/sglang/blob/03e04b23312a1c6f5f16cd4dfffd530fb4210a65/python/sglang/srt/managers/router/model_rpc.py#L587, when `--num-gpus > 1`. It will raise `AttributeError: Can't pickle local object 'start_model_process.._init_service'` as the `_init_service` function is a local variable, which cannot be pickled. Making `_init_service` a global function will resolve this problem. But I haven't tried the launcher provided by sglang itself.

Reproduce the benchmark using the provided fastchat.serve.sglang_worker command and inspect fastchat/serve/sglang_worker.py at line 164, then separately investigate sglang's python/sglang/srt/managers/router/model_rpc.py at line 587 with --num-gpus greater than 1. Done means both reported exceptions are resolved or the issue is split with clear ownership and reproduction results.

Written by the indexing model from the issue text.