Echtzeitsysteme / Echtzeitsysteme/java-refactoring-ttc

Execution time measurements

Open
#38 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
0
Forks
0
PR merge metrics
No merged PRs in 30d

Description

The case description demands solution authors to put the execution times of all test cases as measured by ARTE into the solution descriptions. However, all test cases are very small toy examples. When I run `execute --all` the worst execution time I get for my solution is ~0.7 seconds for `pub_pum3_1`. However, when I execute just that using `execute --test pub_pum3_1` it finishes in ~0.02 which is about the time needed for the other test cases, too. So that single longer execution time seems to be some GC hiccup which only happens when executing all tests in a sequence or something alike.

Well, long story short: I doubt the execution times measured for these toy examples is significant. Can't you provide some performance test cases, e.g., a base case with 10 classes with 5 pullable and 5 non-pullable members each, and then further cases where the base case is duplicated several times to get some larger models? (Like with the train benchmark case where the models double in size.)

Also, I think as soon as you use something like JaMoPP or Modisco, the execution times are probably dominated by parsing, reference resolution, and serialization, i.e., all the stuff which has actually nothing to do with the actual refactorings. So an execution sequence like `parseJava -> transformToPG -> 1000 times: refactor -> serializeAsJava` would be more meaningful with respect to execution time than the sequences containing at most two refactorings which ARTE does right now.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.