alteryx / alteryx/evalml

Add utilities to compare unit test runtimes between two different runs

Offen
#2,453 2 Kommentare 1 Reaktion 1 zugewiesene Person Beansprucht von @asniyaz Auf GitHub ansehen
spike task testing
Vorherrschende Sprache
Python
Sterne
850
Forks
96
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

After #2448 is merged, we will have artifacts for each job which includes the runtime of each test run. I had previously manually written a XML parser to read in two of these files and compare the runtime of each test in the two different runs. It could be useful to store this in the evalml repo for the team to access easily and use, to help debug or spot huge increases in runtime for our unit tests.

I've attached my original python code to this issue; it was hardcoded for my sake but hopefully it's a good starting point for whoever picks this issue up!

Extra bonus: it could also be super duper cool if we could, during a job, calculate the difference in time it took to run on that branch vs the current artifact on main and raise a warning / flag if the runtime significantly increased.
```
import csv
import requests
import xml.etree.ElementTree as ET



def parse_xml(xmlfile):
tree = ET.parse(xmlfile)
root = tree.getroot()
results = {}
for test_case in root.findall('testcase'):
attributes = test_case.attrib
name = attributes['name']
time = attributes['time']
if float(time) > 1:
results[name] = time
return results

def compare_results(main_results, other_results):
timing_results_dict = {}
for test_case_name in main_results:
try:
time_diff = float(other_results[test_case_name]) - float(main_results[test_case_name])
timing_results_dict[test_case_name] = time_diff
except KeyError:
continue
return timing_results_dict

def main():
main_results = parse_xml('main.xml')
woodwork_results = parse_xml('woodwork_36.xml')
time_diffs = compare_results(main_results, woodwork_results)
# for test_case in time_diffs:
# print(test_case, time_diffs[test_case])
sorted_diffs = sorted(time_diffs.items(), key=lambda x:x[1])
for diff in sorted_diffs:
print (diff)

if __name__ == "__main__":

# calling main function
main()
```

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.