Ability to support large datasets
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 6.5k
- Forks
- 787
- PR merge metrics
- No merged PRs in 30d
Description
SUMMARY
StackStorm actions+workflows works great on small datasets. However, as datasets grow StackStorm begins to slow down significantly. For example, i'm working with some data that is a few 100KB in size and StackStorm workflows runs in about 60 seconds, the majority of the time is spent reading/writing data from MongoDB. Also, this puts a large load on the st2rulesengine when triggers for these actions come through, as it needs to query the database for the large datasets.
This feature request is to support large datasets in a more performant way (desired outcome). How this is achieved is flexible.
ISSUE TYPE
- Feature Idea
EXPECTED RESULTS
Handling medium-large datasets in workflows has minimal performance impact.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by profiling the actions and workflows that read and write larger datasets in MongoDB, then trace how st2rulesengine queries those datasets when triggers arrive. Compare the current behavior with the expected result of minimal performance impact for medium-large workflow datasets; the issue leaves the implementation and relevant files open.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- mongodb, python
- Domain
- backend, databases, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100