flosell / flosell/trailscraper

Supports streams of events instead of large JSON documents to make select/generate more efficient

Open
#112 0 comments 0 reactions 0 assignees View on GitHub
enhancement generate-feature performance select-feature
Dominant language
Python
Stars
838
Forks
38
Avg merge
2m
Merged PRs (30d)
14

Description

At the moment, `trailscraper select` produces a full JSON document with the same format as it would appear in a normal CloudTrail JSON file. `trailscraper generate` reads this format.

This is very flexible as it composes well with other tools that might be used to deal with this kind of data. However, it takes a while longer (and possibly generates more overhead) since each step needs to completely finish their operation before handing it off the the next step.
If we'd output the selected events one by one as we found them, generate could already get to work on them.

Next step:
1. Prototype if this would actually make a significant improvement to decide if it's worth complicating the usage over this

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.