StackStorm / StackStorm/st2

Large query to /executions fails in mongodb

Open
#3,111 8 comments 0 reactions 1 assignee View on GitHub

@Kami is already working on this.

Since Dec 14, 2016.

performance question stale
Dominant language
Python
Stars
6.5k
Forks
787
PR merge metrics
No merged PRs in 30d

Description

The indexes added in 2.1 helped with a similar error but now I'm getting the following using production data that did not reproduce using 2k test executions.

My use case is that I need to query all the parent executions (status, action name, arguments, start/stop time) in the past 24 hours where the action name matches a substring. This allows me to throttle/deny additional requests from users based on a set of policies.

2016-12-13 22:03:54,425 140362311938768 INFO hooks [-] e02fb208-4202-4fee-bd77-1e4d7fe356d8 -  GET /executions with filters={u'limit': u'2000', u'exclude_attributes': u'result,trigger_instance', u'timestamp_gt': u'2016-12-12T22:03:54.403458Z', u'parent': u'null'} (remote_addr='127.0.0.1',method='GET',filters={u'exclude_attributes': u'result,trigger_instance', u'limit': u'2000', u'parent': u'null', u'timestamp_gt': u'2016-12-12T22:03:54.403458Z'},request_id='e02fb208-4202-4fee-bd77-1e4d7fe356d8',path='/executions')
[..]
2016-12-13 22:03:54,430 140362311938768 INFO resource [-] GET all /executions with filters={'start_timestamp__gt': datetime.datetime(2016, 12, 12, 22, 3, 54, 403458, tzinfo=tzutc()), 'order_by': ['+start_timestamp', 'action.ref'], 'parent': u'null'} (offset=0,limit=2000,filters={'start_timestamp__gt': datetime.datetime(2016, 12, 12, 22, 3, 54, 403458, tzinfo=tzutc()), 'order_by': ['+start_timestamp', 'action.ref'], 'parent': u'null'},sort=[])
2016-12-13 22:03:54,578 140362311938768 ERROR hooks [-] API call failed: errmsg: "Sort operation used more than the maximum 33554432 bytes of RAM. Add an index, or specify a smaller limit."
Traceback (most recent call last):
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/pecan/core.py", line 631, in __call__
    self.invoke_controller(controller, args, kwargs, state)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/pecan/core.py", line 531, in invoke_controller
    result = controller(*args, **kwargs)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/rbac/decorators.py", line 61, in func_wrapper
    return func(*args, **kwargs)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/models/api/base.py", line 284, in callfunction
    raise e
OperationFailure: errmsg: "Sort operation used more than the maximum 33554432 bytes of RAM. Add an index, or specify a smaller limit." (_exception_data={'_OperationFailure__code': 96, '_OperationFailure__details': {u'ok': 0.0, u'code': 96, u'waitedMS': 0L, u'errmsg': u'errmsg: "Sort operation used more than the maximum 33554432 bytes of RAM. Add an index, or specify a smaller limit."'}},_exception_class='OperationFailure',_exception_message='errmsg: "Sort operation used more than the maximum 33554432 bytes of RAM. Add an index, or specify a smaller limit."')

Is it worth increasing internalQueryExecMaxBlockingStortBytes or is there a missing index that would help here? Or is there an index that is not being updated often enough?

Thanks!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.