microsoft / microsoft/pai

Worker node's resources don't show on Webportal

Open
#4,640 4 comments 0 reactions 1 assignee View on GitHub

@Binyang2014 is already working on this.

Since Jun 24, 2020.

0.14.0 raised by customer
Dominant language
JavaScript
Stars
2.7k
Forks
554
Avg merge
20h 42m
Merged PRs (30d)
14

Description

OpenPAI Environment:

  • OpenPAI version: 0.14.0
  • OS (e.g. from /etc/os-release): Ubuntu 18.04

The main problem:
I've faced a problem second time with worker nodes when I can't see their resources on the dashboard (Grafana) on webportal and on Virtual Clusters page (I see IP of worker node, but all resources show me as 0). However, I can see any information on "Hardware" page and I see all resources there. Also, I see it in "total nodes" not in "worker nodes" (Grafana). In addition, everything is in a good condition in k8s.

They are together had the same bug after the situation when their memory was over and they didn't have swap. After this docker restarted and containers for services were in "created" status, not in "UP".

What I tried to do:

  • I tried to remove and add the wrong machine in the cluster.
  • I tried to delete all dedicated clusters where this machine was.
  • I tried to restart daemon and docker service after removing this machine from the cluster.
  • I deleted all data in /datastorage directory.
  • I added swap for this wrong machine and did anything which I noticed before.
  • I tried to fix crashed etcd node after when I added it in the cluster using python paictl.py machine etcd-fix -p /path/to/config -l machine-list.yaml and had some wrong output:

image

  • I solved this problem for the first time only after deleting all machines and services. However, I can't do it now, because I want to save all information from "Jobs" and I don't want to clean configuration.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.