OpenBMB / OpenBMB/XAgent

gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>

Open
#383 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
8.5k
Forks
905
No merged PRs in 30d

Description

Issue Description / 问题描述

Please provide a detailed description of the error or issue you encountered. / 请详细描述您遇到的错误或问题。

gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>

Steps to Reproduce / 复现步骤

Please provide the specific steps to reproduce the error. / 请提供复现错误的具体步骤。

docker compose up

Expected Behavior / 预期行为

Describe the behavior you expected to see. / 请描述您期望的正确行为。
No Error.

Environment / 环境信息

  • Operating System / 操作系统：Windows
  • Python Version / Python 版本：3.10
  • Other Relevant Information / 其他相关信息：In WSL2 with docker-desktop.

Error Screenshots or Logs / 错误截图或日志

If possible, please provide relevant screenshots or logs of the error. / 如果可能，请提供相关的错误截图或日志文件。

Additional Notes / 其他备注

If you have any additional information or notes, please add them here. / 如果有其他补充信息，请在此处添加。

➜  XAgent git:(main) ✗ docker compose up
[+] Running 5/0
 ⠿ Container xagent-ToolServerNode-1  Created                                                                                                                                  0.0s
 ⠿ Container xagent-xagent-mysql-1    Running                                                                                                                                  0.0s
 ⠿ Container xagent-db-1              Running                                                                                                                                  0.0s
 ⠿ Container xagent-xagent-redis-1    Running                                                                                                                                  0.0s
 ⠿ Container XAgent-Server            Running                                                                                                                                  0.0s
Attaching to XAgent-Server, xagent-ToolServerManager-1, xagent-ToolServerNode-1, xagent-db-1, xagent-xagent-mysql-1, xagent-xagent-redis-1
xagent-ToolServerManager-1  | [2024-01-28 10:16:11 +0000] [1] [INFO] Starting gunicorn 21.2.0
xagent-ToolServerManager-1  | [2024-01-28 10:16:11 +0000] [1] [INFO] Listening at: http://0.0.0.0:8080 (1)
xagent-ToolServerManager-1  | [2024-01-28 10:16:11 +0000] [1] [INFO] Using worker: uvicorn.workers.UvicornWorker
xagent-ToolServerManager-1  | [2024-01-28 10:16:11 +0000] [7] [INFO] Booting worker with pid: 7
xagent-ToolServerManager-1  | [2024-01-28 10:16:11 +0000] [8] [INFO] Booting worker with pid: 8
xagent-ToolServerNode-1     | Traceback (most recent call last):
xagent-ToolServerNode-1     |   File "/usr/local/bin/uvicorn", line 8, in <module>
xagent-ToolServerNode-1     |     sys.exit(main())
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
xagent-ToolServerNode-1     |     return self.main(*args, **kwargs)
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1078, in main
xagent-ToolServerNode-1     |     rv = self.invoke(ctx)
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
xagent-ToolServerNode-1     |     return ctx.invoke(self.callback, **ctx.params)
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 783, in invoke
xagent-ToolServerNode-1     |     return __callback(*args, **kwargs)
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/site-packages/uvicorn/main.py", line 416, in main
xagent-ToolServerNode-1     |     run(
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/site-packages/uvicorn/main.py", line 587, in run
xagent-ToolServerNode-1     |     server.run()
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/site-packages/uvicorn/server.py", line 61, in run
xagent-ToolServerNode-1     |     return asyncio.run(self.serve(sockets=sockets))
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/asyncio/runners.py", line 44, in run
xagent-ToolServerNode-1     |     return loop.run_until_complete(main)
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
xagent-ToolServerNode-1     |     return future.result()
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/site-packages/uvicorn/server.py", line 68, in serve
xagent-ToolServerNode-1     |     config.load()
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/site-packages/uvicorn/config.py", line 467, in load
xagent-ToolServerNode-1     |     self.loaded_app = import_from_string(self.app)
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/site-packages/uvicorn/importer.py", line 21, in import_from_string
xagent-ToolServerNode-1     |     module = importlib.import_module(module_str)
xagent-ToolServerNode-1     |   File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
xagent-ToolServerNode-1     |     return _bootstrap._gcd_import(name[level:], package, level)
xagent-ToolServerNode-1     |   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
xagent-ToolServerNode-1     |   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
xagent-ToolServerNode-1     |   File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
xagent-ToolServerNode-1     |   File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
xagent-ToolServerNode-1     |   File "<frozen importlib._bootstrap_external>", line 883, in exec_module
xagent-ToolServerNode-1     |   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
xagent-ToolServerNode-1     |   File "/app/main.py", line 13, in <module>
xagent-ToolServerNode-1     |     from config import CONFIG,logger
xagent-ToolServerNode-1     |   File "/app/config.py", line 73, in <module>
xagent-ToolServerNode-1     |     CONFIG = NodeConfig()
xagent-ToolServerNode-1     |   File "/app/config.py", line 31, in __init__
xagent-ToolServerNode-1     |     self.cfg:Dict = yaml.load(open(config_file_path, "r", encoding="utf-8").read(), Loader=yaml.FullLoader)
xagent-ToolServerNode-1     | FileNotFoundError: [Errno 2] No such file or directory: './assets/config/node.yml'
xagent-ToolServerNode-1 exited with code 1
xagent-ToolServerManager-1  | [2024-01-28 10:16:12 +0000] [7] [ERROR] Exception in worker process
xagent-ToolServerManager-1  | Traceback (most recent call last):
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 609, in spawn_worker
xagent-ToolServerManager-1  |     worker.init_process()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/uvicorn/workers.py", line 66, in init_process
xagent-ToolServerManager-1  |     super(UvicornWorker, self).init_process()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/workers/base.py", line 134, in init_process
xagent-ToolServerManager-1  |     self.load_wsgi()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
xagent-ToolServerManager-1  |     self.wsgi = self.app.wsgi()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/app/base.py", line 67, in wsgi
xagent-ToolServerManager-1  |     self.callable = self.load()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
xagent-ToolServerManager-1  |     return self.load_wsgiapp()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
xagent-ToolServerManager-1  |     return util.import_app(self.app_uri)
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/util.py", line 371, in import_app
xagent-ToolServerManager-1  |     mod = importlib.import_module(module)
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
xagent-ToolServerManager-1  |     return _bootstrap._gcd_import(name[level:], package, level)
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap_external>", line 883, in exec_module
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
xagent-ToolServerManager-1  |   File "/app/main.py", line 12, in <module>
xagent-ToolServerManager-1  |     from config import CONFIG,logger,MANAGER_ID
xagent-ToolServerManager-1  |   File "/app/config.py", line 65, in <module>
xagent-ToolServerManager-1  |     CONFIG = ManagerConfig()
xagent-ToolServerManager-1  |   File "/app/config.py", line 27, in __init__
xagent-ToolServerManager-1  |     self.cfg:Dict = yaml.load(open(config_file_path,"r",encoding="utf-8").read(), Loader=yaml.FullLoader)
xagent-ToolServerManager-1  | FileNotFoundError: [Errno 2] No such file or directory: './assets/config/manager.yml'
xagent-ToolServerManager-1  | [2024-01-28 10:16:12 +0000] [8] [ERROR] Exception in worker process
xagent-ToolServerManager-1  | Traceback (most recent call last):
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 609, in spawn_worker
xagent-ToolServerManager-1  |     worker.init_process()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/uvicorn/workers.py", line 66, in init_process
xagent-ToolServerManager-1  |     super(UvicornWorker, self).init_process()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/workers/base.py", line 134, in init_process
xagent-ToolServerManager-1  |     self.load_wsgi()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
xagent-ToolServerManager-1  |     self.wsgi = self.app.wsgi()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/app/base.py", line 67, in wsgi
xagent-ToolServerManager-1  |     self.callable = self.load()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
xagent-ToolServerManager-1  |     return self.load_wsgiapp()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
xagent-ToolServerManager-1  |     return util.import_app(self.app_uri)
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/util.py", line 371, in import_app
xagent-ToolServerManager-1  |     mod = importlib.import_module(module)
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
xagent-ToolServerManager-1  |     return _bootstrap._gcd_import(name[level:], package, level)
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap_external>", line 883, in exec_module
xagent-ToolServerManager-1  |   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
xagent-ToolServerManager-1  |   File "/app/main.py", line 12, in <module>
xagent-ToolServerManager-1  |     from config import CONFIG,logger,MANAGER_ID
xagent-ToolServerManager-1  |   File "/app/config.py", line 65, in <module>
xagent-ToolServerManager-1  |     CONFIG = ManagerConfig()
xagent-ToolServerManager-1  |   File "/app/config.py", line 27, in __init__
xagent-ToolServerManager-1  |     self.cfg:Dict = yaml.load(open(config_file_path,"r",encoding="utf-8").read(), Loader=yaml.FullLoader)
xagent-ToolServerManager-1  | FileNotFoundError: [Errno 2] No such file or directory: './assets/config/manager.yml'
xagent-ToolServerManager-1  | [2024-01-28 10:16:12 +0000] [8] [INFO] Worker exiting (pid: 8)
xagent-ToolServerManager-1  | [2024-01-28 10:16:12 +0000] [7] [INFO] Worker exiting (pid: 7)
xagent-ToolServerManager-1  | [2024-01-28 10:16:12 +0000] [1] [ERROR] Worker (pid:7) exited with code 3
xagent-ToolServerManager-1  | [2024-01-28 10:16:12 +0000] [1] [ERROR] Worker (pid:8) exited with code 3
xagent-ToolServerManager-1  | Traceback (most recent call last):
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 209, in run
xagent-ToolServerManager-1  |     self.sleep()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 360, in sleep
xagent-ToolServerManager-1  |     ready = select.select([self.PIPE[0]], [], [], 1.0)
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 242, in handle_chld
xagent-ToolServerManager-1  |     self.reap_workers()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 530, in reap_workers
xagent-ToolServerManager-1  |     raise HaltServer(reason, self.WORKER_BOOT_ERROR)
xagent-ToolServerManager-1  | gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>
xagent-ToolServerManager-1  | 
xagent-ToolServerManager-1  | During handling of the above exception, another exception occurred:
xagent-ToolServerManager-1  | 
xagent-ToolServerManager-1  | Traceback (most recent call last):
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 662, in kill_worker
xagent-ToolServerManager-1  |     os.kill(pid, sig)
xagent-ToolServerManager-1  | ProcessLookupError: [Errno 3] No such process
xagent-ToolServerManager-1  | 
xagent-ToolServerManager-1  | During handling of the above exception, another exception occurred:
xagent-ToolServerManager-1  | 
xagent-ToolServerManager-1  | Traceback (most recent call last):
xagent-ToolServerManager-1  |   File "/usr/local/bin/gunicorn", line 8, in <module>
xagent-ToolServerManager-1  |     sys.exit(run())
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py", line 67, in run
xagent-ToolServerManager-1  |     WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/app/base.py", line 236, in run
xagent-ToolServerManager-1  |     super().run()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/app/base.py", line 72, in run
xagent-ToolServerManager-1  |     Arbiter(self).run()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 229, in run
xagent-ToolServerManager-1  |     self.halt(reason=inst.reason, exit_status=inst.exit_status)
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 342, in halt
xagent-ToolServerManager-1  |     self.stop()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 393, in stop
xagent-ToolServerManager-1  |     self.kill_workers(sig)
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 652, in kill_workers
xagent-ToolServerManager-1  |     self.kill_worker(pid, sig)
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 662, in kill_worker
xagent-ToolServerManager-1  |     os.kill(pid, sig)
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 242, in handle_chld
xagent-ToolServerManager-1  |     self.reap_workers()
xagent-ToolServerManager-1  |   File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 530, in reap_workers
xagent-ToolServerManager-1  |     raise HaltServer(reason, self.WORKER_BOOT_ERROR)
xagent-ToolServerManager-1  | gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>
xagent-ToolServerManager-1 exited with code 1

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running docker compose up and inspect the failing main.py entry points and config.py initializers in the ToolServer Node and Manager containers. Check how ./assets/config/node.yml and ./assets/config/manager.yml are expected to be provided, then verify the containers start without the reported FileNotFoundError and Gunicorn worker failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, docker-compose, python, yaml
Domain
backend, devops, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.