Worker node's resources don't show on Webportal
@Binyang2014 is already working on this.
Since Jun 24, 2020.
- Dominant language
- JavaScript
- Stars
- 2.7k
- Forks
- 554
- Avg merge
- 20h 42m
- Merged PRs (30d)
- 14
Description
OpenPAI Environment:
- OpenPAI version: 0.14.0
- OS (e.g. from /etc/os-release): Ubuntu 18.04
The main problem:
I've faced a problem second time with worker nodes when I can't see their resources on the dashboard (Grafana) on webportal and on Virtual Clusters page (I see IP of worker node, but all resources show me as 0). However, I can see any information on "Hardware" page and I see all resources there. Also, I see it in "total nodes" not in "worker nodes" (Grafana). In addition, everything is in a good condition in k8s.
They are together had the same bug after the situation when their memory was over and they didn't have swap. After this docker restarted and containers for services were in "created" status, not in "UP".
What I tried to do:
- I tried to remove and add the wrong machine in the cluster.
- I tried to delete all dedicated clusters where this machine was.
- I tried to restart daemon and docker service after removing this machine from the cluster.
- I deleted all data in
/datastoragedirectory. - I added swap for this wrong machine and did anything which I noticed before.
- I tried to fix crashed etcd node after when I added it in the cluster using
python paictl.py machine etcd-fix -p /path/to/config -l machine-list.yamland had some wrong output:

- I solved this problem for the first time only after deleting all machines and services. However, I can't do it now, because I want to save all information from "Jobs" and I don't want to clean configuration.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.