github / github/codeql

CodeQL CLI may be slow to run when the codeql.zip is extracted at $HOME

未關閉
#13,208 1 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
acknowledged question
主要語言
CodeQL
星號
10.1k
分支
2.1k
平均合併
2 天 15 小時
30 天內合併 PR
141

描述

In the [Getting started with the CodeQL CLI](https://docs.github.com/code-security/codeql-cli/using-the-codeql-cli/getting-started-with-the-codeql-cli) documentation, it instructs the user to extract the downloaded zip archive, but doesn't advise where this should be:

> ### 2. Extract the zip archive
>
> For Linux, Windows, and macOS users (version 10.14 "Mojave", and earlier) simply extract the zip archive.
>
> [...]
>
> ### 3. Launch `codeql`
>
> Once extracted, you can run CodeQL processes by running the `codeql` executable in a couple of ways:
>
> * By executing `/codeql/codeql`, where `` is the folder where you extracted the CodeQL CLI package.
>
>[...]

If the user chooses to extract in a directory tree which may contain a very large number of files and subdirectories, such as their home directory, then upon running any `codeql` command that resolves extensions, all of these subdirectories will be searched for files with certain filename patterns, which may take a very long time.

For example (output is from running `kill -3` on the `java` process):

```
"pool-1-thread-1" #22 prio=5 os_prio=0 cpu=40821.78ms elapsed=40.94s tid=0x00007f3dec64a460 nid=0x1bd12 runnable [0x00007f3dac3e3000]
java.lang.Thread.State: RUNNABLE
at sun.nio.fs.UnixNativeDispatcher.lstat0(java.base@17.0.7/Native Method)
at sun.nio.fs.UnixNativeDispatcher.lstat(java.base@17.0.7/UnixNativeDispatcher.java:308)
at sun.nio.fs.UnixFileAttributes.get(java.base@17.0.7/UnixFileAttributes.java:72)
at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(java.base@17.0.7/UnixFileAttributeViews.java:52)
at sun.nio.fs.UnixFileSystemProvider.readAttributes(java.base@17.0.7/UnixFileSystemProvider.java:148)
at sun.nio.fs.LinuxFileSystemProvider.readAttributes(java.base@17.0.7/LinuxFileSystemProvider.java:99)
at java.nio.file.Files.readAttributes(java.base@17.0.7/Files.java:1851)
at java.nio.file.FileTreeWalker.getAttributes(java.base@17.0.7/FileTreeWalker.java:220)
at java.nio.file.FileTreeWalker.visit(java.base@17.0.7/FileTreeWalker.java:277)
at java.nio.file.FileTreeWalker.next(java.base@17.0.7/FileTreeWalker.java:374)
at java.nio.file.Files.walkFileTree(java.base@17.0.7/Files.java:2845)
at com.semmle.frontend.packs.DiskFileSystem.directChildren(DiskFileSystem.java:56)
at com.semmle.frontend.packs.glob.PathMatchTreeWalker.pushStackEntry(PathMatchTreeWalker.java:345)
at com.semmle.frontend.packs.glob.PathMatchTreeWalker.matchDirectory(PathMatchTreeWalker.java:205)
at com.semmle.frontend.packs.glob.PathMatchTreeWalker.walk(PathMatchTreeWalker.java:121)
at com.semmle.frontend.packs.glob.PathMatcher.walkFileTree(PathMatcher.java:270)
at com.semmle.frontend.packs.ScanByManifests.scanTree(ScanByManifests.java:354)
at com.semmle.frontend.packs.ScanByManifests.scanForPacksAndWorkspaces(ScanByManifests.java:154)
at com.semmle.frontend.packs.QlPackResolver.scanForPacksAndWorkspaces(QlPackResolver.java:107)
at com.semmle.frontend.packs.QlPackContainer.computePacksAndWorkspaces(QlPackContainer.java:119)
at com.semmle.frontend.packs.QlPackContainer.lambda$new$0(QlPackContainer.java:38)
at com.semmle.frontend.packs.QlPackContainer$$Lambda$145/0x0000000800d8dd70.apply(Unknown Source)
at com.semmle.api.compilation.CachedOperation.get(CachedOperation.java:91)
- locked <0x0000000591e21578> (a java.lang.Object)
at com.semmle.frontend.packs.QlPackContainer.findPacksByName(QlPackContainer.java:69)
at com.semmle.frontend.packs.QlPackSearchContext.findPacksByName(QlPackSearchContext.java:698)
at com.semmle.frontend.packs.QlPackSearchContext.findPackByName(QlPackSearchContext.java:604)
at com.semmle.frontend.packs.QlPackSearchContext.findPackByVersion(QlPackSearchContext.java:414)
at com.semmle.frontend.packs.QlPackSearchContext.lambda$findAllPackDependencies$19(QlPackSearchContext.java:509)
at com.semmle.frontend.packs.QlPackSearchContext$$Lambda$248/0x0000000800db9560.get(Unknown Source)
at java.util.Optional.orElseGet(java.base@17.0.7/Optional.java:364)
at com.semmle.frontend.packs.QlPackSearchContext.lambda$findAllPackDependencies$20(QlPackSearchContext.java:495)
at com.semmle.frontend.packs.QlPackSearchContext$$Lambda$246/0x0000000800db90d0.apply(Unknown Source)
at java.util.Optional.map(java.base@17.0.7/Optional.java:260)
at com.semmle.frontend.packs.QlPackSearchContext.findAllPackDependencies(QlPackSearchContext.java:467)
at com.semmle.cli2.resolve.ResolveExtensionsCommand.resolvePackDependencies(ResolveExtensionsCommand.java:267)
at com.semmle.cli2.resolve.ResolveExtensionsCommand.executeJSON(ResolveExtensionsCommand.java:70)
at com.semmle.cli2.resolve.ResolveExtensionsCommand.executeJSON(ResolveExtensionsCommand.java:43)
at com.semmle.cli2.picocli.SimpleJsonSubcommand.executeInsistingOnJSONWithExistingMessages(SimpleJsonSubcommand.java:130)
at com.semmle.cli2.picocli.SimpleJsonSubcommand$$Lambda$66/0x0000000800cf0a88.apply(Unknown Source)
at com.semmle.cli2.picocli.SubcommandCommon.withCompilationMessages(SubcommandCommon.java:412)
at com.semmle.cli2.picocli.SimpleJsonSubcommand.executeInsistingOnJSON(SimpleJsonSubcommand.java:119)
at com.semmle.cli2.picocli.PlumbingRunner.call(PlumbingRunner.java:53)
at com.semmle.cli2.picocli.SubcommandCommon.callPlumbingInProcess(SubcommandCommon.java:160)
at com.semmle.cli2.test.CodeqlTestDir$ExtensionValuesCache.calculateExtensions(CodeqlTestDir.java:63)
at com.semmle.cli2.test.CodeqlTestDir$ExtensionValuesCache.get(CodeqlTestDir.java:53)
- locked <0x0000000591e21a18> (a java.util.concurrent.ConcurrentHashMap)
at com.semmle.cli2.test.CodeqlTestDir.resolveExtensions(CodeqlTestDir.java:199)
at com.semmle.cli2.test.CodeqlTestDir$$Lambda$118/0x0000000800d83990.run(Unknown Source)
at java.util.concurrent.CompletableFuture$UniRun.tryFire(java.base@17.0.7/CompletableFuture.java:787)
at java.util.concurrent.CompletableFuture.postComplete(java.base@17.0.7/CompletableFuture.java:510)
at java.util.concurrent.CompletableFuture$AsyncRun.run(java.base@17.0.7/CompletableFuture.java:1810)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.7/ThreadPoolExecutor.java:1136)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.7/ThreadPoolExecutor.java:635)
at java.lang.Thread.run(java.base@17.0.7/Thread.java:833)
```

The root cause seems to be that method `com.semmle.cli2.resolve.ResolveExtensionsCommand.resolvePackDependencies` calls `com.semmle.frontend.packs.QlPackSearchContext.findAllPackDependencies` with `useLegacyResolution` set to true, which ends up conducting a search of the parent of the `codeql` directory.

Please could you either change this behaviour, or make the documentation clearer about where the extraction root for `codeql.zip` should be to avoid this issue?

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。