github / github/codeql

CodeQL CLI may be slow to run when the codeql.zip is extracted at $HOME

未关闭
#13,208 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
acknowledged question
主要语言
CodeQL
星标
10.1k
派生
2.1k
平均合并
2 天 15 小时
30 天内合并 PR
141

描述

In the [Getting started with the CodeQL CLI](https://docs.github.com/code-security/codeql-cli/using-the-codeql-cli/getting-started-with-the-codeql-cli) documentation, it instructs the user to extract the downloaded zip archive, but doesn't advise where this should be:

> ### 2. Extract the zip archive
>
> For Linux, Windows, and macOS users (version 10.14 "Mojave", and earlier) simply extract the zip archive.
>
> [...]
>
> ### 3. Launch `codeql`
>
> Once extracted, you can run CodeQL processes by running the `codeql` executable in a couple of ways:
>
> * By executing `/codeql/codeql`, where `` is the folder where you extracted the CodeQL CLI package.
>
>[...]

If the user chooses to extract in a directory tree which may contain a very large number of files and subdirectories, such as their home directory, then upon running any `codeql` command that resolves extensions, all of these subdirectories will be searched for files with certain filename patterns, which may take a very long time.

For example (output is from running `kill -3` on the `java` process):

```
"pool-1-thread-1" #22 prio=5 os_prio=0 cpu=40821.78ms elapsed=40.94s tid=0x00007f3dec64a460 nid=0x1bd12 runnable [0x00007f3dac3e3000]
java.lang.Thread.State: RUNNABLE
at sun.nio.fs.UnixNativeDispatcher.lstat0(java.base@17.0.7/Native Method)
at sun.nio.fs.UnixNativeDispatcher.lstat(java.base@17.0.7/UnixNativeDispatcher.java:308)
at sun.nio.fs.UnixFileAttributes.get(java.base@17.0.7/UnixFileAttributes.java:72)
at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(java.base@17.0.7/UnixFileAttributeViews.java:52)
at sun.nio.fs.UnixFileSystemProvider.readAttributes(java.base@17.0.7/UnixFileSystemProvider.java:148)
at sun.nio.fs.LinuxFileSystemProvider.readAttributes(java.base@17.0.7/LinuxFileSystemProvider.java:99)
at java.nio.file.Files.readAttributes(java.base@17.0.7/Files.java:1851)
at java.nio.file.FileTreeWalker.getAttributes(java.base@17.0.7/FileTreeWalker.java:220)
at java.nio.file.FileTreeWalker.visit(java.base@17.0.7/FileTreeWalker.java:277)
at java.nio.file.FileTreeWalker.next(java.base@17.0.7/FileTreeWalker.java:374)
at java.nio.file.Files.walkFileTree(java.base@17.0.7/Files.java:2845)
at com.semmle.frontend.packs.DiskFileSystem.directChildren(DiskFileSystem.java:56)
at com.semmle.frontend.packs.glob.PathMatchTreeWalker.pushStackEntry(PathMatchTreeWalker.java:345)
at com.semmle.frontend.packs.glob.PathMatchTreeWalker.matchDirectory(PathMatchTreeWalker.java:205)
at com.semmle.frontend.packs.glob.PathMatchTreeWalker.walk(PathMatchTreeWalker.java:121)
at com.semmle.frontend.packs.glob.PathMatcher.walkFileTree(PathMatcher.java:270)
at com.semmle.frontend.packs.ScanByManifests.scanTree(ScanByManifests.java:354)
at com.semmle.frontend.packs.ScanByManifests.scanForPacksAndWorkspaces(ScanByManifests.java:154)
at com.semmle.frontend.packs.QlPackResolver.scanForPacksAndWorkspaces(QlPackResolver.java:107)
at com.semmle.frontend.packs.QlPackContainer.computePacksAndWorkspaces(QlPackContainer.java:119)
at com.semmle.frontend.packs.QlPackContainer.lambda$new$0(QlPackContainer.java:38)
at com.semmle.frontend.packs.QlPackContainer$$Lambda$145/0x0000000800d8dd70.apply(Unknown Source)
at com.semmle.api.compilation.CachedOperation.get(CachedOperation.java:91)
- locked <0x0000000591e21578> (a java.lang.Object)
at com.semmle.frontend.packs.QlPackContainer.findPacksByName(QlPackContainer.java:69)
at com.semmle.frontend.packs.QlPackSearchContext.findPacksByName(QlPackSearchContext.java:698)
at com.semmle.frontend.packs.QlPackSearchContext.findPackByName(QlPackSearchContext.java:604)
at com.semmle.frontend.packs.QlPackSearchContext.findPackByVersion(QlPackSearchContext.java:414)
at com.semmle.frontend.packs.QlPackSearchContext.lambda$findAllPackDependencies$19(QlPackSearchContext.java:509)
at com.semmle.frontend.packs.QlPackSearchContext$$Lambda$248/0x0000000800db9560.get(Unknown Source)
at java.util.Optional.orElseGet(java.base@17.0.7/Optional.java:364)
at com.semmle.frontend.packs.QlPackSearchContext.lambda$findAllPackDependencies$20(QlPackSearchContext.java:495)
at com.semmle.frontend.packs.QlPackSearchContext$$Lambda$246/0x0000000800db90d0.apply(Unknown Source)
at java.util.Optional.map(java.base@17.0.7/Optional.java:260)
at com.semmle.frontend.packs.QlPackSearchContext.findAllPackDependencies(QlPackSearchContext.java:467)
at com.semmle.cli2.resolve.ResolveExtensionsCommand.resolvePackDependencies(ResolveExtensionsCommand.java:267)
at com.semmle.cli2.resolve.ResolveExtensionsCommand.executeJSON(ResolveExtensionsCommand.java:70)
at com.semmle.cli2.resolve.ResolveExtensionsCommand.executeJSON(ResolveExtensionsCommand.java:43)
at com.semmle.cli2.picocli.SimpleJsonSubcommand.executeInsistingOnJSONWithExistingMessages(SimpleJsonSubcommand.java:130)
at com.semmle.cli2.picocli.SimpleJsonSubcommand$$Lambda$66/0x0000000800cf0a88.apply(Unknown Source)
at com.semmle.cli2.picocli.SubcommandCommon.withCompilationMessages(SubcommandCommon.java:412)
at com.semmle.cli2.picocli.SimpleJsonSubcommand.executeInsistingOnJSON(SimpleJsonSubcommand.java:119)
at com.semmle.cli2.picocli.PlumbingRunner.call(PlumbingRunner.java:53)
at com.semmle.cli2.picocli.SubcommandCommon.callPlumbingInProcess(SubcommandCommon.java:160)
at com.semmle.cli2.test.CodeqlTestDir$ExtensionValuesCache.calculateExtensions(CodeqlTestDir.java:63)
at com.semmle.cli2.test.CodeqlTestDir$ExtensionValuesCache.get(CodeqlTestDir.java:53)
- locked <0x0000000591e21a18> (a java.util.concurrent.ConcurrentHashMap)
at com.semmle.cli2.test.CodeqlTestDir.resolveExtensions(CodeqlTestDir.java:199)
at com.semmle.cli2.test.CodeqlTestDir$$Lambda$118/0x0000000800d83990.run(Unknown Source)
at java.util.concurrent.CompletableFuture$UniRun.tryFire(java.base@17.0.7/CompletableFuture.java:787)
at java.util.concurrent.CompletableFuture.postComplete(java.base@17.0.7/CompletableFuture.java:510)
at java.util.concurrent.CompletableFuture$AsyncRun.run(java.base@17.0.7/CompletableFuture.java:1810)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.7/ThreadPoolExecutor.java:1136)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.7/ThreadPoolExecutor.java:635)
at java.lang.Thread.run(java.base@17.0.7/Thread.java:833)
```

The root cause seems to be that method `com.semmle.cli2.resolve.ResolveExtensionsCommand.resolvePackDependencies` calls `com.semmle.frontend.packs.QlPackSearchContext.findAllPackDependencies` with `useLegacyResolution` set to true, which ends up conducting a search of the parent of the `codeql` directory.

Please could you either change this behaviour, or make the documentation clearer about where the extraction root for `codeql.zip` should be to avoid this issue?

贡献指南

打开贡献指南

调研方向

Start with the linked “Getting started with the CodeQL CLI” page and review the “Extract the zip archive” and “Launch codeql” sections. Determine whether the intended resolution is clearer extraction guidance or a CLI behavior change; the documentation should prevent extraction under large directory trees while keeping the launch instructions accurate.

由索引模型根据 Issue 内容生成。

评估

领域
cli, documentation
Issue 类型
缺陷
难度
3/5
预计耗时
1-2 天
活跃度
停滞
描述清晰度
基本清楚
新手友好度
42/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。