Performance issue with classpath scanning

What problem are you trying to solve?

When running OpenRewrite on larger code bases, with many recipes enabled, I would normally assume it uses all the available cores while it runs. It doesn't. It rather produces a pattern of short spikes (of almost full CPU usage) followed by longer parts of ~30% CPU usage on my multi-core Windows machine.

Attaching a profiler makes me think this is entirely related to the classpath scanning. What I can see

• the classpath library methods accumulate hours of runtime
• a huge amount of the classpath scanning runtime is caused by calls to the file system (which is extremely slow on Windows compared to Linux, especially if some antivirus tool is enabled)

I therefore believe that at least on Windows the runtime is dominated by ongoing repetitions of

• slow classpath scanning
• short recipe execution

Describe the solution you'd like

This could potentially be improved by splitting the scanning and the analysis into separate runnables executed by 2 different threadpools (where a finished scanning thread would start a recipe thread). In theory this would lead to the classpath scanning running almost all the time (limited by the file system performance and the antivirus stuff), and refactorings being triggered as often as possible in between.

This would still not lead to full CPU usage probably, but at least to higher CPU usage.

Have you considered any alternatives or workarounds?

I have not yet understood whether or not the classpath scanning could eventually be reduced as such. That would of course be an even better solution.

Are you interested in contributing this feature to OpenRewrite?

Maybe. Not sure if I can actually rework the code in this manner. Currently I see this issue more as a discussion about potential improvements.

The issue names no files, tests, or entry points. Start by reproducing the classpath-scanning and recipe-execution pattern on Windows with a profiler, then trace the classpath library calls and filesystem work. Done would require a defined, measured performance improvement and agreement on whether scanning reduction or concurrent execution is the intended direction.