Re-visit estimates on thread counts, process limits, file descriptor limits, etc.

Newer systems may have hyper threading on and core counts are now much larger than before. Estimates for the number of threads daemons should have to handle communications (for example NameNode, DataNode, etc.) were previously estimates based on node count. These estimates may now be out of date and need to be calculated differently. Revisit calculations for these estimates.

In addition, max number of tasks (such as in Hadoop or Spark) may also need to be re-estimated. While in the past it may have been common to have 8-24 cores, with hyper threading 48-64 is not that unreasonable. The trade off of more threads/tasks may no longer be balanced in favor of big data applications. Re-consider how max threads/tasks per node is determined.

Locate the scripts or entry points that calculate daemon thread counts, task limits, and file descriptor limits, then trace the assumptions based on node and core counts. Compare those calculations with the Hadoop and Spark use cases described in the issue. Done means the estimates and per-node limits have been reconsidered for larger and hyper-threaded systems, with the affected behavior validated.