testfiles: consider reevaluating all files to ensure runtime requirement is met
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 6.2k
- Forks
- 726
- Avg merge
- 11d 11h
- Merged PRs (30d)
- 7
Description
https://github.com/mandiant/capa-testfiles/pull/322 modified multiple files, some of which were committed a long time ago, which triggered the runtime check to execute. This identified that at least three of the files do not meet our current runtime requirements:
```
ERROR capa.tests.data: capa ran for 308.1699028015137 check_runtimes.py:61
seconds on
b5f0524e69b3a3cf636c7ac366ca57bf5e3a8fdc8a9f01caf1
96c611a7918a87.elf_, please provide a different
sample so we can test more quickly
ERROR capa.tests.data: capa ran for 374.43834686279297 check_runtimes.py:61
seconds on 92d8ea10ea30e8b534334a1c9857a455.exe_,
please provide a different sample so we can test
more quickly
ERROR capa.tests.data: capa ran for 275.35956621170044 check_runtimes.py:61
seconds on
559efe9f3d4864910d5c5edbaadd3972c6dfc3ba887c48aa67
28b0093898563f.dll_, please provide a different
sample so we can test more quickly
```
These three files alone account for **15 minutes** of runtime. The runtime check only executes for the files modified/added in a given pull request, so there are likely other files that have flown under the radar. IIRC this is only an issue when a thorough test is manually triggered one or more times before a new release. We should reevaluate all test files and replace those that no longer meet our runtime requirements if thorough test runtime becomes a concern.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the runtime check at check_runtimes.py:61 and inspect the three samples named in the issue, then determine how the full test-file set is selected for thorough runs. Run the thorough test to identify other slow samples and replace any that exceed the current runtime requirement. Done means the reevaluated files meet the runtime limit without changing the test purpose.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- performance, reverse-engineering, testing-qa
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100