microsoft / microsoft/pai

The service cannot be started after installation.

Open
#5,576 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
JavaScript
Stars
2.7k
Forks
554
Avg merge
20h 42m
Merged PRs (30d)
14

Description

maybe duel to docker-cache service do not start after /bin/bash quick-start-service.sh

But the job cannot be started. This is the log of kubelet, it keeps outputting the image pull completion. The openpai system did not output any logs.

csip@csip-090:~$ sudo systemctl status kubelet.service 
● kubelet.service - Kubernetes Kubelet Server
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2021-04-27 13:19:30 CST; 4min 39s ago
     Docs: https://github.com/GoogleCloudPlatform/kubernetes
 Main PID: 173276 (kubelet)
    Tasks: 0
   Memory: 32.5M
      CPU: 1.107s
   CGroup: /system.slice/kubelet.service
           ‣ 173276 /usr/local/bin/kubelet --logtostderr=true --v=2 --node-ip=172.17.175.90 --hostname-override=csip-090 --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --config=/etc/kubernetes/kubelet-config.yaml --kubeconfig=/etc/kubernetes/kubelet.conf 

Apr 27 13:23:31 csip-090 kubelet[173276]: I0427 13:23:31.097675  173276 kube_docker_client.go:342] Pulling image "openpai/standard:python_3.6-pytorch_1.2.0-gpu": "7b872974e97c: Pull complete "
Apr 27 13:23:31 csip-090 kubelet[173276]: I0427 13:23:31.776293  173276 setters.go:73] Using node IP: "172.17.175.90"
Apr 27 13:23:41 csip-090 kubelet[173276]: I0427 13:23:41.097618  173276 kube_docker_client.go:342] Pulling image "openpai/standard:python_3.6-pytorch_1.2.0-gpu": "7b872974e97c: Pull complete "
Apr 27 13:23:41 csip-090 kubelet[173276]: I0427 13:23:41.788918  173276 setters.go:73] Using node IP: "172.17.175.90"
Apr 27 13:23:51 csip-090 kubelet[173276]: I0427 13:23:51.097632  173276 kube_docker_client.go:342] Pulling image "openpai/standard:python_3.6-pytorch_1.2.0-gpu": "7b872974e97c: Pull complete "
Apr 27 13:23:51 csip-090 kubelet[173276]: I0427 13:23:51.382029  173276 kubelet_getters.go:177] status for pod nginx-proxy-csip-090 updated to {Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2021-04-27 11:48:40 +0800 CST  } {Ready True 0001-01-01 00:00:00 +000
Apr 27 13:23:51 csip-090 kubelet[173276]: I0427 13:23:51.662060  173276 endpoint.go:111] State pushed for device plugin github.com/fuse
Apr 27 13:23:51 csip-090 kubelet[173276]: I0427 13:23:51.800339  173276 setters.go:73] Using node IP: "172.17.175.90"
Apr 27 13:24:01 csip-090 kubelet[173276]: I0427 13:24:01.097620  173276 kube_docker_client.go:342] Pulling image "openpai/standard:python_3.6-pytorch_1.2.0-gpu": "7b872974e97c: Pull complete "
Apr 27 13:24:01 csip-090 kubelet[173276]: I0427 13:24:01.813667  173276 setters.go:73] Using node IP: "172.17.175.90"

Originally posted by @siaimes in https://github.com/microsoft/pai/issues/5445#issuecomment-827327246

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing quick-start-service.sh and the Docker cache service startup path, then inspect the reported kubelet status and repeated image-pull messages. Reproduce the installation and startup sequence to determine whether the cache service or another component prevents the system from starting. Done means the service starts successfully and OpenPAI produces its expected logs.

Written by the indexing model from the issue text.

Assessment

Tech stack
bash, docker, kubernetes
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.