alteryx / alteryx/evalml

Add utilities to compare unit test runtimes between two different runs

Aperta
#2,453 2 commenti 1 reazione 1 assegnatario Rivendicata da @asniyaz Vedi su GitHub
spike task testing
Lingua principale
Python
Stelle
850
Fork
96
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

After #2448 is merged, we will have artifacts for each job which includes the runtime of each test run. I had previously manually written a XML parser to read in two of these files and compare the runtime of each test in the two different runs. It could be useful to store this in the evalml repo for the team to access easily and use, to help debug or spot huge increases in runtime for our unit tests.

I've attached my original python code to this issue; it was hardcoded for my sake but hopefully it's a good starting point for whoever picks this issue up!

Extra bonus: it could also be super duper cool if we could, during a job, calculate the difference in time it took to run on that branch vs the current artifact on main and raise a warning / flag if the runtime significantly increased.
```
import csv
import requests
import xml.etree.ElementTree as ET



def parse_xml(xmlfile):
tree = ET.parse(xmlfile)
root = tree.getroot()
results = {}
for test_case in root.findall('testcase'):
attributes = test_case.attrib
name = attributes['name']
time = attributes['time']
if float(time) > 1:
results[name] = time
return results

def compare_results(main_results, other_results):
timing_results_dict = {}
for test_case_name in main_results:
try:
time_diff = float(other_results[test_case_name]) - float(main_results[test_case_name])
timing_results_dict[test_case_name] = time_diff
except KeyError:
continue
return timing_results_dict

def main():
main_results = parse_xml('main.xml')
woodwork_results = parse_xml('woodwork_36.xml')
time_diffs = compare_results(main_results, woodwork_results)
# for test_case in time_diffs:
# print(test_case, time_diffs[test_case])
sorted_diffs = sorted(time_diffs.items(), key=lambda x:x[1])
for diff in sorted_diffs:
print (diff)

if __name__ == "__main__":

# calling main function
main()
```

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.