alteryx / alteryx/evalml

Add utilities to compare unit test runtimes between two different runs

未关闭
#2,453 2 条评论 1 个 reaction 已指派 1 人 已被 @asniyaz 认领 在 GitHub 查看
spike task testing
主要语言
Python
星标
850
派生
96
PR 合并指标
30 天内没有已合并 PR

描述

After #2448 is merged, we will have artifacts for each job which includes the runtime of each test run. I had previously manually written a XML parser to read in two of these files and compare the runtime of each test in the two different runs. It could be useful to store this in the evalml repo for the team to access easily and use, to help debug or spot huge increases in runtime for our unit tests.

I've attached my original python code to this issue; it was hardcoded for my sake but hopefully it's a good starting point for whoever picks this issue up!

Extra bonus: it could also be super duper cool if we could, during a job, calculate the difference in time it took to run on that branch vs the current artifact on main and raise a warning / flag if the runtime significantly increased.
```
import csv
import requests
import xml.etree.ElementTree as ET



def parse_xml(xmlfile):
tree = ET.parse(xmlfile)
root = tree.getroot()
results = {}
for test_case in root.findall('testcase'):
attributes = test_case.attrib
name = attributes['name']
time = attributes['time']
if float(time) > 1:
results[name] = time
return results

def compare_results(main_results, other_results):
timing_results_dict = {}
for test_case_name in main_results:
try:
time_diff = float(other_results[test_case_name]) - float(main_results[test_case_name])
timing_results_dict[test_case_name] = time_diff
except KeyError:
continue
return timing_results_dict

def main():
main_results = parse_xml('main.xml')
woodwork_results = parse_xml('woodwork_36.xml')
time_diffs = compare_results(main_results, woodwork_results)
# for test_case in time_diffs:
# print(test_case, time_diffs[test_case])
sorted_diffs = sorted(time_diffs.items(), key=lambda x:x[1])
for diff in sorted_diffs:
print (diff)

if __name__ == "__main__":

# calling main function
main()
```

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。