autoreload: Improve performance with many files (watchdog plugin)

Currently when running tornado with autoreload, especially on VM's, the CPU usage is extremely high. For example:

We are running our development environment with docker. It loads around 400 files. It does a stat on every file and it is causing 100% cpu load on one of my cores. I am not using the latest mac book (2015) but my colleagues are using more recent models, which also suffer from high cpu usage. The load gets higher, on more cores, especially when running a few microservices in containers.

The CPU load is caused by the way autoreload is implemented. It just does a 'stat' on all imported python files every N seconds. There are much better ways to do that, for example: mechanisms like inotify, which is available on linux. Or use something like watchman, which supports many modes and has proven to work with a huge amounts of files. Watchman is used in javascript land quite a lot and can cope with file changes in huge projects. It can watch (parts of) node_modules/ for example. Additionally django is using watchman now as well.

Note: Inotify is not cross platform, stat is working on all platforms i guess. Different platforms have different better solutions. Watchman covers that, but introduces a lot of dependencies.

What would be nice is if there was a way of choosing/configuring autoreload mechanisms. The "old" stat autoreload plugin can then be used by default. Custom autoreload plugins can then be implemented by developers, or different plugins can be shipped together with tornado.

So the proposed solution is two fold:

• Implement plugin structure
• Implement the stat reloader as a plugin

What i hope that happens afterward by the community

• Implement os specific reloaders, starting with inotify for linux
• Implement watchman reloader, which covers everything.

Currently web.py already has a configuration for enabling autoreload (self.settings.get("autoreload")). What could be added is an option which defaults to the stat plugin loader ("autoreload_stat") or if a setting is available tries to load that plugin.
self.settings.get("autoreload_plugin", default="autoreload_stat")

Side note: When going through the tickets i also discovered this issue, which can be solved with a (3rd party) watchman plugin i think.

Start by reading Tornado's autoreload implementation and the autoreload setting referenced in tornado/web.py. Define how a plugin structure would select the existing stat reloader while leaving room for platform-specific reloaders such as inotify or Watchman. Done means the stat reloader remains available and a configurable plugin mechanism is specified and implemented.