nfsserver monitor reports healthy service, but clients are unable to mount

NFS client suddenly throws below errors and no long mount the filesystem, even after repeated attempts (including rebooting client node).

```
May 30 10:37:42 client-2 kernel: nfs: server 10.0.1.210 not responding, timed out
May 30 10:37:47 client-2 kernel: nfs: server 10.0.1.210 not responding, timed out
May 30 10:37:52 client-2 kernel: nfs: server 10.0.1.210 not responding, timed out
```

Mount from client node fails with 2 types of errors:

_**Error1: portmap query failed: RPC: Remote system error - Connection timed out**_

```
[opc@client-2 ~]$ sudo mount -v /mnt/nfs
mount.nfs: trying text-based options 'vers=3,bg,timeo=100,ac,actimeo=120,nocto,rsize=1048576,wsize=1048576,nolock,local_lock=none,proto=tcp,sec=sys,addr=10.0.1.210'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 10.0.1.210 prog 100003 vers 3 prot TCP port 2049
mount.nfs: portmap query failed: RPC: Remote system error - Connection timed out
mount.nfs: backgrounding "10.0.1.210:/mnt/nfsshare/exports"
mount.nfs: mount options: "rw,noatime,nodiratime,vers=3,bg,timeo=100,ac,actimeo=120,nocto,rsize=1048576,wsize=1048576,nolock,local_lock=none,proto=tcp,sec=sys,_netdev"
```

_**Error2: portmap query failed: RPC: Timed out**_

```
[opc@client-2 ~]$ sudo mount -v /mnt/nfs
mount.nfs: trying text-based options 'vers=3,bg,timeo=100,ac,actimeo=120,nocto,rsize=1048576,wsize=1048576,nolock,local_lock=none,proto=tcp,sec=sys,addr=10.0.1.210'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: portmap query failed: RPC: Timed out
mount.nfs: backgrounding "10.0.1.210:/mnt/nfsshare/exports"
mount.nfs: mount options: "rw,noatime,nodiratime,vers=3,bg,timeo=100,ac,actimeo=120,nocto,rsize=1048576,wsize=1048576,nolock,local_lock=none,proto=tcp,sec=sys,_netdev"

```

Troubleshooting:
On File server node - NFS service monitoring shows service is fine and in fact, if we run a NFS client on the file server itself, it mounts correctly. So we investigated why NFS service monitor says "all good" , but client cannot mount it and found this check which works on server locally (used by NFS service monitor), but fails from client node.

https://github.com/ClusterLabs/resource-agents/blob/master/heartbeat/nfsserver#L726

```
[opc@storage-server-1 ~]$ rpcinfo -t 10.0.2.233 100024
program 100024 version 1 ready and waiting

[opc@client-2 ~]$ rpcinfo -t 10.0.1.210 100024
rpcinfo: RPC: Port mapper failure - Timed out
program 100024 is not available
```

Seems like, the monitoring requires additional checks to ensure NFS clients can connect and mount and if not, return monitoring failed.

Is there a workaround , anyone can suggest, so the IO error doesn't happen ?

Start at heartbeat/nfsserver around line 726 and compare the local rpcinfo check with the client-side failures shown in the report. Reproduce the check against the NFS server from a client node; done means the monitor detects the reported connectivity problem instead of reporting the service healthy.

Written by the indexing model from the issue text.