document node termination behavior

Feature Request

Our docs do not describe what to do when a node is terminated (essentially disk is gone and cannot be recovered). In this case the operator will automatically add a new member to the statefulset after a brief period. The old member will still exist in a pending un-scheduleable state.

The user can choose to leave it as is for a while. But it will block upgrades, so although it isn't an emergency, the correct course of action is to deal with it soon. Recovering without adjusting the statefulset can end up with one more pod then they wanted.

Recovery requires

• manually remove the store from PD
  • find the id with kubectl describe -n namespace tidbclusters.pingcap.com db
  • delete it with pd-ctl store delete or otherwise using the PD API DELETE /store/{storeID}
• delete the PVC of the pod
• delete the pod

This situation helps motivate our development of an improved statefulset

No documentation file or test is named. Locate the operator documentation covering terminated nodes and statefulsets, then verify the listed kubectl, pd-ctl or PD API, PVC, and pod-removal steps; done means the recovery procedure and its upgrade implications are clearly documented.