Question about large spark application's being throttled

Nobody has claimed this yet.

Hi I'm kinda new to dr elephant and when I was deploying and testing it on my machines, I found large spark application logs (large than 100MB by default) will be ignored and won't show up in the UI due to the throttle behavior. There might be something I missed, but based on the code, does it mean dr elephant will skip all spark applications whose size is larger than `eventLogSizeLimitMb` (by default 100MB)? Is this how `dataCollection.throttle()` expected to behave? If my understanding is wrong, can someone tell me how the throttle works for large spark applications? If my understanding is right, is there any remarkable bottleneck on dr elephant for large spark applications? I think it's easy for a spark application log to go beyond several GB, and how dr elephant's gonna solve it? Thanks in advance for any helps and suggestions!

Start by tracing dataCollection.throttle() and the eventLogSizeLimitMb configuration. Confirm whether oversized Spark event logs are skipped, identify any existing tests or documentation for that path, and define the expected behavior for multi-GB logs before proposing work.

Written by the indexing model from the issue text.