ModelEngine-Group / ModelEngine-Group/DataMate

datamate-frontend 冷启动时可能连不上 datamate-gateway;datamate-runtime 在 Docker 默认共享内存下可能 OOM

Open Beginner friendly
#529 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
368
Forks
46
Avg merge
35m
Merged PRs (30d)
4

Description

问题一:服务启动顺序缺失

deployment/docker/datamate/docker-compose.yml 里 datamate-frontend 的
depends_on 列出了 datamate-backend 和 datamate-backend-python,但没有
datamate-gateway。冷启动 docker-compose up 时,frontend 可能在 gateway
就绪之前就先启动了,导致 frontend 请求 gateway 失败,直到手动重启才恢复。

问题二:Ray 在默认 Docker 共享内存下 OOM

datamate-runtime 跑的是基于 Ray 的流水线,但没有设置 shm_size。Docker 默认
的 /dev/shm(64MB)远低于 Ray object store 实际需要的量——这是 Ray 部署里
一个众所周知的、会导致高负载下 worker 崩溃的原因。

建议修复方向
  • 给 datamate-frontend 的 depends_on 加上 datamate-gateway。
  • 给 datamate-runtime 加 shm_size: "10gb",并调整
    RAY_DEFAULT_OBJECT_STORE_MEMORY_PROPORTION。

我已经准备好 compose 文件的修改,会提一个 PR。

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Open deployment/docker/datamate/docker-compose.yml and inspect the datamate-frontend and datamate-runtime service definitions first. Verify the dependency and shared-memory settings, then run docker compose config and a cold-start compose run; done means frontend can start with datamate-gateway available and the runtime has the requested Ray memory configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, docker-compose
Domain
devops, infrastructure
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.