Add utilities to compare unit test runtimes between two different runs
- Lingua principale
- Python
- Stelle
- 850
- Fork
- 96
- Metriche di merge delle PR
- Nessuna PR unita negli ultimi 30g
Descrizione
After #2448 is merged, we will have artifacts for each job which includes the runtime of each test run. I had previously manually written a XML parser to read in two of these files and compare the runtime of each test in the two different runs. It could be useful to store this in the evalml repo for the team to access easily and use, to help debug or spot huge increases in runtime for our unit tests.
I've attached my original python code to this issue; it was hardcoded for my sake but hopefully it's a good starting point for whoever picks this issue up!
Extra bonus: it could also be super duper cool if we could, during a job, calculate the difference in time it took to run on that branch vs the current artifact on main and raise a warning / flag if the runtime significantly increased.
```
import csv
import requests
import xml.etree.ElementTree as ET
def parse_xml(xmlfile):
tree = ET.parse(xmlfile)
root = tree.getroot()
results = {}
for test_case in root.findall('testcase'):
attributes = test_case.attrib
name = attributes['name']
time = attributes['time']
if float(time) > 1:
results[name] = time
return results
def compare_results(main_results, other_results):
timing_results_dict = {}
for test_case_name in main_results:
try:
time_diff = float(other_results[test_case_name]) - float(main_results[test_case_name])
timing_results_dict[test_case_name] = time_diff
except KeyError:
continue
return timing_results_dict
def main():
main_results = parse_xml('main.xml')
woodwork_results = parse_xml('woodwork_36.xml')
time_diffs = compare_results(main_results, woodwork_results)
# for test_case in time_diffs:
# print(test_case, time_diffs[test_case])
sorted_diffs = sorted(time_diffs.items(), key=lambda x:x[1])
for diff in sorted_diffs:
print (diff)
if __name__ == "__main__":
# calling main function
main()
```
Guida per i contributori
Apri la guida per i contributori
Valutazione
Questa issue non è ancora stata valutata.