CodeQL CLI may be slow to run when the codeql.zip is extracted at $HOME
- Dominant language
- CodeQL
- Stars
- 10.1k
- Forks
- 2.1k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 141
Description
In the [Getting started with the CodeQL CLI](https://docs.github.com/code-security/codeql-cli/using-the-codeql-cli/getting-started-with-the-codeql-cli) documentation, it instructs the user to extract the downloaded zip archive, but doesn't advise where this should be:
> ### 2. Extract the zip archive
>
> For Linux, Windows, and macOS users (version 10.14 "Mojave", and earlier) simply extract the zip archive.
>
> [...]
>
> ### 3. Launch `codeql`
>
> Once extracted, you can run CodeQL processes by running the `codeql` executable in a couple of ways:
>
> * By executing `/codeql/codeql`, where `` is the folder where you extracted the CodeQL CLI package.
>
>[...]
If the user chooses to extract in a directory tree which may contain a very large number of files and subdirectories, such as their home directory, then upon running any `codeql` command that resolves extensions, all of these subdirectories will be searched for files with certain filename patterns, which may take a very long time.
For example (output is from running `kill -3` on the `java` process):
```
"pool-1-thread-1" #22 prio=5 os_prio=0 cpu=40821.78ms elapsed=40.94s tid=0x00007f3dec64a460 nid=0x1bd12 runnable [0x00007f3dac3e3000]
java.lang.Thread.State: RUNNABLE
at sun.nio.fs.UnixNativeDispatcher.lstat0(java.base@17.0.7/Native Method)
at sun.nio.fs.UnixNativeDispatcher.lstat(java.base@17.0.7/UnixNativeDispatcher.java:308)
at sun.nio.fs.UnixFileAttributes.get(java.base@17.0.7/UnixFileAttributes.java:72)
at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(java.base@17.0.7/UnixFileAttributeViews.java:52)
at sun.nio.fs.UnixFileSystemProvider.readAttributes(java.base@17.0.7/UnixFileSystemProvider.java:148)
at sun.nio.fs.LinuxFileSystemProvider.readAttributes(java.base@17.0.7/LinuxFileSystemProvider.java:99)
at java.nio.file.Files.readAttributes(java.base@17.0.7/Files.java:1851)
at java.nio.file.FileTreeWalker.getAttributes(java.base@17.0.7/FileTreeWalker.java:220)
at java.nio.file.FileTreeWalker.visit(java.base@17.0.7/FileTreeWalker.java:277)
at java.nio.file.FileTreeWalker.next(java.base@17.0.7/FileTreeWalker.java:374)
at java.nio.file.Files.walkFileTree(java.base@17.0.7/Files.java:2845)
at com.semmle.frontend.packs.DiskFileSystem.directChildren(DiskFileSystem.java:56)
at com.semmle.frontend.packs.glob.PathMatchTreeWalker.pushStackEntry(PathMatchTreeWalker.java:345)
at com.semmle.frontend.packs.glob.PathMatchTreeWalker.matchDirectory(PathMatchTreeWalker.java:205)
at com.semmle.frontend.packs.glob.PathMatchTreeWalker.walk(PathMatchTreeWalker.java:121)
at com.semmle.frontend.packs.glob.PathMatcher.walkFileTree(PathMatcher.java:270)
at com.semmle.frontend.packs.ScanByManifests.scanTree(ScanByManifests.java:354)
at com.semmle.frontend.packs.ScanByManifests.scanForPacksAndWorkspaces(ScanByManifests.java:154)
at com.semmle.frontend.packs.QlPackResolver.scanForPacksAndWorkspaces(QlPackResolver.java:107)
at com.semmle.frontend.packs.QlPackContainer.computePacksAndWorkspaces(QlPackContainer.java:119)
at com.semmle.frontend.packs.QlPackContainer.lambda$new$0(QlPackContainer.java:38)
at com.semmle.frontend.packs.QlPackContainer$$Lambda$145/0x0000000800d8dd70.apply(Unknown Source)
at com.semmle.api.compilation.CachedOperation.get(CachedOperation.java:91)
- locked <0x0000000591e21578> (a java.lang.Object)
at com.semmle.frontend.packs.QlPackContainer.findPacksByName(QlPackContainer.java:69)
at com.semmle.frontend.packs.QlPackSearchContext.findPacksByName(QlPackSearchContext.java:698)
at com.semmle.frontend.packs.QlPackSearchContext.findPackByName(QlPackSearchContext.java:604)
at com.semmle.frontend.packs.QlPackSearchContext.findPackByVersion(QlPackSearchContext.java:414)
at com.semmle.frontend.packs.QlPackSearchContext.lambda$findAllPackDependencies$19(QlPackSearchContext.java:509)
at com.semmle.frontend.packs.QlPackSearchContext$$Lambda$248/0x0000000800db9560.get(Unknown Source)
at java.util.Optional.orElseGet(java.base@17.0.7/Optional.java:364)
at com.semmle.frontend.packs.QlPackSearchContext.lambda$findAllPackDependencies$20(QlPackSearchContext.java:495)
at com.semmle.frontend.packs.QlPackSearchContext$$Lambda$246/0x0000000800db90d0.apply(Unknown Source)
at java.util.Optional.map(java.base@17.0.7/Optional.java:260)
at com.semmle.frontend.packs.QlPackSearchContext.findAllPackDependencies(QlPackSearchContext.java:467)
at com.semmle.cli2.resolve.ResolveExtensionsCommand.resolvePackDependencies(ResolveExtensionsCommand.java:267)
at com.semmle.cli2.resolve.ResolveExtensionsCommand.executeJSON(ResolveExtensionsCommand.java:70)
at com.semmle.cli2.resolve.ResolveExtensionsCommand.executeJSON(ResolveExtensionsCommand.java:43)
at com.semmle.cli2.picocli.SimpleJsonSubcommand.executeInsistingOnJSONWithExistingMessages(SimpleJsonSubcommand.java:130)
at com.semmle.cli2.picocli.SimpleJsonSubcommand$$Lambda$66/0x0000000800cf0a88.apply(Unknown Source)
at com.semmle.cli2.picocli.SubcommandCommon.withCompilationMessages(SubcommandCommon.java:412)
at com.semmle.cli2.picocli.SimpleJsonSubcommand.executeInsistingOnJSON(SimpleJsonSubcommand.java:119)
at com.semmle.cli2.picocli.PlumbingRunner.call(PlumbingRunner.java:53)
at com.semmle.cli2.picocli.SubcommandCommon.callPlumbingInProcess(SubcommandCommon.java:160)
at com.semmle.cli2.test.CodeqlTestDir$ExtensionValuesCache.calculateExtensions(CodeqlTestDir.java:63)
at com.semmle.cli2.test.CodeqlTestDir$ExtensionValuesCache.get(CodeqlTestDir.java:53)
- locked <0x0000000591e21a18> (a java.util.concurrent.ConcurrentHashMap)
at com.semmle.cli2.test.CodeqlTestDir.resolveExtensions(CodeqlTestDir.java:199)
at com.semmle.cli2.test.CodeqlTestDir$$Lambda$118/0x0000000800d83990.run(Unknown Source)
at java.util.concurrent.CompletableFuture$UniRun.tryFire(java.base@17.0.7/CompletableFuture.java:787)
at java.util.concurrent.CompletableFuture.postComplete(java.base@17.0.7/CompletableFuture.java:510)
at java.util.concurrent.CompletableFuture$AsyncRun.run(java.base@17.0.7/CompletableFuture.java:1810)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.7/ThreadPoolExecutor.java:1136)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.7/ThreadPoolExecutor.java:635)
at java.lang.Thread.run(java.base@17.0.7/Thread.java:833)
```
The root cause seems to be that method `com.semmle.cli2.resolve.ResolveExtensionsCommand.resolvePackDependencies` calls `com.semmle.frontend.packs.QlPackSearchContext.findAllPackDependencies` with `useLegacyResolution` set to true, which ends up conducting a search of the parent of the `codeql` directory.
Please could you either change this behaviour, or make the documentation clearer about where the extraction root for `codeql.zip` should be to avoid this issue?
Contributor guide
Research direction
Start with the linked “Getting started with the CodeQL CLI” page and review the “Extract the zip archive” and “Launch codeql” sections. Determine whether the intended resolution is clearer extraction guidance or a CLI behavior change; the documentation should prevent extraction under large directory trees while keeping the launch instructions accurate.
Written by the indexing model from the issue text.
Assessment
- Domain
- cli, documentation
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100