Validation Checklist

Validation Checklist

The validation checklist after deploy.

Alert

• Test item completed. @Binyang2014

1. Check if the new bits trigger any alert!

Version

• Test item completed. @hzy46

1. Version beside the "Feedback" menu item in web portal, can be modified in "version" field of "src/webportal/package.json"

User Management

• Test item completed. @abuccts
  Username and password can be found in services-configuration config file: restserver->default-pai-admin-{username, password}

1. Login:
   1. Correct username/password => login success
   2. Incorrect username/password => login fails with corresponding error message
2. Logout:
   1. Login user can logout
3. Change own password
4. Admin user:
   1. List users
   2. Add users
   3. Update users VC, password

Virtual Clusters

• Test item completed. @abuccts

1. Displayed VC names, capacity, memory, CPU, GPU consist with configuration
2. “View Jobs” can be accessed and content correct
3. Add a new vc
4. Delete a vc

Dashboard view

• Test item completed. @Binyang2014

1. Landing page dashboard
   1. The Grafana page shown correctly (e.g., no "N/A", "no data point")
2. K8s dashboard: (admin user only)
   1. All pods and daemon sets are running well
   2. User can view pod's log and exec command from kubernetes dashboard
3. Service view: (admin user only)
   1. Node name, role and count consist with configuration file, PAI_TASK_ROLE_COUNT
   2. Service status green
4. Hardware view: (admin user only)
   1. IP address, machine name correct
   2. No ‘N/A’ value in CPU/Memory/GPU/GPU Mem/Disk/Ethernet status
5. Grafana view:
   1. Cluster metrics
   2. Node metrics
   3. Job level metrics (go to jobs page, and select one running job then click "Go to Job Metrics Page")
   4. Task level metrics (go to jobs page, and select one running job then click "Go to Job Metrics Page")
   5. Task role level metrics (go to jobs page, and select one running job then click "Go to Job Metrics Page")

Jobs Page

• Test item completed. @yiyione

1. Job list
2. Job search
3. Job sorting according to each column
4. Job can be stopped by job owner and button disabled to other users
5. View job config
6. SSH infos
7. Job logs can be accessed during job running and after job finishes.

Submit job

• Test item completed. @abuccts

1. Submit form can import job config file and fill in corresponding fields.
2. Incorrect json format jobs cannot be accepted.
3. Non-login user has no permission to submit job
4. Submit jobs to different VCs with/without permission and check result
5. Submit sample cntk/tensorflow/etc. jobs and check whether jobs can finish successfully.

Azure RDMA

• Test item completed. @abuccts

1. Based on this knowledge, please ensure all worker machines in your cluster is rdma-capable and they are all in the same available set.
2. Build the intel-mpi bench marking image. https://github.com/Microsoft/pai/tree/v0.14.0/examples/azure-rdma-inte-mpi-benchmark-with-horovod-image
3. Make sure AZ_RDMA is enabled in rest-server env variables
4. Submit the job through the webportal with the json in this link.Note: Please change the "image" field with yours image url.

Storage

• Test item completed. @Binyang2014

1. Make sure teamwise storage is well set. Go to job submit page, open Data section on the right. Check if "DEFAULT" teamwise config exists and selected
2. Add job name, use any docker image, fill Command with mkdir -p /mnt/data/${PAI_USERNAME} && echo ${PAI_JOB_NAME} > /mnt/data/${PAI_USERNAME}/${PAI_JOB_NAME}.txt
3. Open folder on teamwise storage machine, enter PAI user's folder and check if new file with job name created.

Client

• Test item completed. @yiyione

1. Test NNI tool and try to submit job from NNI to OpenPAI cluster and make sure the job can be submitted and run correctly.
2. In Openmind tool launch a job on OpenPAI and check the job can be run correctly.
3. Submit jobs in VSCode Client.

Marketplace

• Test item completed. @debuggy
  // TODO

Test Scripts

• Test item completed. @yiyione
  // TODO

Start by reviewing the validation checklist, src/webportal/package.json, the services-configuration settings, and the Azure RDMA example linked in the issue. Run through the deployment, portal, Kubernetes, dashboard, job, storage, and client checks. Done means the checklist items are completed and the Marketplace and Test Scripts TODOs are addressed.