Running tile-reduce accross containers

The current internal workings of tile-reduce make it very good for running on a single machine, but due to an issue with require('os').cpus.length, which doesn't accurately report the number of CPU's available within a docker container, tile-reduce can be very resource hungry when running in a container.

Rather than working with a single-machine/process-forking model for distributing work, could we write a version of tile-reduce that can run with workers as their own short-lived containers that communicate to a master container using HTTP or TCP? For use in ECS, we could use https://github.com/mapbox/ecs-watchbot to manage the orchestration of running new containers on an AWS ECS cluster. We could possibly create affordances for other orchestration tools (like mesos or kubernetes) if other people would like them.

cc/ @nickcordella @mourner @tcql @rclark

Start by reviewing tile-reduce's current single-machine, process-forking model and how require('os').cpus.length behaves inside Docker containers. Define the master/worker communication and orchestration approach, then verify that short-lived worker containers can process jobs through ECS and the proposed HTTP or TCP connection.