Set hard limit for maximum API response size

Right now we have no hard limits for API response size.

This means if user requests a lot of data (e.g. GET /v1/executions`` API call with many very large executions in the database), it will take a long time for API to convert from pymongo to mongoengine objects and then serialize those objects as JSON for end-user consumption.

It's very likely that this will also block API gunicorn worker (DB reads are async, but object conversion and JSON serialization is not) and possibly cause a time out. This is problematic because our default st2api configuration file starts st2api server with a single gunicorn worker process.

In short term we should do the following:

1. Document this issue and document workarounds (spin more st2api services, increase number of gunicorn workers per service, increase gunicorn worker timeout, switch to StackStorm >= v2.9.0 which includes ?include_attributes improvements so only data which is needed is retrieved).
2. Increase gunicorn worker timeout in a default st2api service manager file

In slightly longer term, I propose following changes:

1. Enforce hard "maximum response" size limit for API responses.

This way we will prevent gunicorn API process from ever being blocked for too long.

There are multiple ways to achieve that, but we need to do that, ideally before retrieving DB models from the database and before serializing it to JSON.

Doing it before serializing it to JSON is easy (after we retrieve DB objects from the database). Doing it before pymongo doing it's internal dict conversion will be a bit more involved.

We will likely need to utilize mongodb Object.bsonsize(db.<collection name>.find({query})) operation to retrieve number of bytes for a particular set of MongoDB documents in a collection.

Then we need to determine "reasonable" limit which API on average sized server can handle. If resulting set is larger than this limit (perhaps also configurable in st2.conf), we should return 400 bad request with a message along the lines of "Requested data set is too large to be processed, please limit the request size by using ?limit=<n>, ?include_attribute=<attributes> and other API query param filters.

This way we will ensure StackStorm API process will never be blocked for too long (you can also look at it as a DDoS / rate limiting protection).

To be on the safe side, we should also add ?force=true or similar query parameter which admins can use to skip that check (of course at their own risk with a big warning in the docs).

2. More WebUI improvements

There is still a lot of place in WebUI to reduce the load on StackStorm API and only retrieve data it needs in a lazy / when needed fashion.

3. Try to offload (blocking) response JSON serialization to eventlet thread pool

Serializing large objects to JSON can be blocking. One way to prevent main event loop from being blocked would be to offload those potentially blocking operations to eventlet thread pool.

I tried this in the past (as a quick improvement), but it's sadly quite involved and I'm not even sure it's possible at all with our current code base (we would need to make the code thread safe which is very hard to do with all the mongoengine objects which cross reference each other, etc.).