alteryx / alteryx/evalml

Add utilities to compare unit test runtimes between two different runs

オープン
#2,453 コメント 2 件 リアクション 1 件 担当者 1 名 @asniyaz が担当を希望しています GitHub で見る
spike task testing
主要言語
Python
スター
850
フォーク
96
PR マージ指標
30日以内にマージされた PR はありません

説明

After #2448 is merged, we will have artifacts for each job which includes the runtime of each test run. I had previously manually written a XML parser to read in two of these files and compare the runtime of each test in the two different runs. It could be useful to store this in the evalml repo for the team to access easily and use, to help debug or spot huge increases in runtime for our unit tests.

I've attached my original python code to this issue; it was hardcoded for my sake but hopefully it's a good starting point for whoever picks this issue up!

Extra bonus: it could also be super duper cool if we could, during a job, calculate the difference in time it took to run on that branch vs the current artifact on main and raise a warning / flag if the runtime significantly increased.
```
import csv
import requests
import xml.etree.ElementTree as ET



def parse_xml(xmlfile):
tree = ET.parse(xmlfile)
root = tree.getroot()
results = {}
for test_case in root.findall('testcase'):
attributes = test_case.attrib
name = attributes['name']
time = attributes['time']
if float(time) > 1:
results[name] = time
return results

def compare_results(main_results, other_results):
timing_results_dict = {}
for test_case_name in main_results:
try:
time_diff = float(other_results[test_case_name]) - float(main_results[test_case_name])
timing_results_dict[test_case_name] = time_diff
except KeyError:
continue
return timing_results_dict

def main():
main_results = parse_xml('main.xml')
woodwork_results = parse_xml('woodwork_36.xml')
time_diffs = compare_results(main_results, woodwork_results)
# for test_case in time_diffs:
# print(test_case, time_diffs[test_case])
sorted_diffs = sorted(time_diffs.items(), key=lambda x:x[1])
for diff in sorted_diffs:
print (diff)

if __name__ == "__main__":

# calling main function
main()
```

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。