Re-consider the APIs for accessing the nodes in the metadata

Explicitly came up during #4007. But, I think this is a common problem in the code as I've thought about this several times, most of the time having hard time.

When we need a list of nodes in the code for a specific purpose, it is pretty complex to choose what we need between many different [APIs](https://github.com/citusdata/citus/blob/96adce77d6683d37e5f4e77324a4da487e4f66ec/src/include/distributed/worker_manager.h#L73-L88).

We had to consider several things:
- Do we need to consider workers with metadata
- Do we need to consider coordinator added as a node in the cluster
- Do we need to consider nodes with distributed table placements (`NodeCanHaveDistTablePlacements `)
- Do we need to consider primary/secondary nodes
- Do we need to consider active nodes
- Do we need to consider Readable nodes

Overall, we should probably re-think how we qualify the nodes in the cluster. Some questions:
- Is there a difference between the worker and the coordinator considering that node both can easily have the placements and/or the metadata?
- Should we have a single API for getting the nodes in the cluster (e.g., `GetNodeList(flags)`) where the caller passes flags that are interesting to that point. We need to think deeper on what these flags should be. We should not directly expose all the fields in `pg_dist_node` as a flag as that'd simply move the problem around.
- We should also consider `nodes with metadata only` as we may implement in the future as *query nodes*.