CodeQL CLI may be slow to run when the codeql.zip is extracted at $HOME
- Ngôn ngữ chính
- CodeQL
- Star
- 10.1k
- Fork
- 2.1k
- Merge trung bình
- 2 ngày 15 giờ
- Pull request đã merge (30 ngày)
- 141
Mô tả
In the [Getting started with the CodeQL CLI](https://docs.github.com/code-security/codeql-cli/using-the-codeql-cli/getting-started-with-the-codeql-cli) documentation, it instructs the user to extract the downloaded zip archive, but doesn't advise where this should be:
> ### 2. Extract the zip archive
>
> For Linux, Windows, and macOS users (version 10.14 "Mojave", and earlier) simply extract the zip archive.
>
> [...]
>
> ### 3. Launch `codeql`
>
> Once extracted, you can run CodeQL processes by running the `codeql` executable in a couple of ways:
>
> * By executing `/codeql/codeql`, where `` is the folder where you extracted the CodeQL CLI package.
>
>[...]
If the user chooses to extract in a directory tree which may contain a very large number of files and subdirectories, such as their home directory, then upon running any `codeql` command that resolves extensions, all of these subdirectories will be searched for files with certain filename patterns, which may take a very long time.
For example (output is from running `kill -3` on the `java` process):
```
"pool-1-thread-1" #22 prio=5 os_prio=0 cpu=40821.78ms elapsed=40.94s tid=0x00007f3dec64a460 nid=0x1bd12 runnable [0x00007f3dac3e3000]
java.lang.Thread.State: RUNNABLE
at sun.nio.fs.UnixNativeDispatcher.lstat0(java.base@17.0.7/Native Method)
at sun.nio.fs.UnixNativeDispatcher.lstat(java.base@17.0.7/UnixNativeDispatcher.java:308)
at sun.nio.fs.UnixFileAttributes.get(java.base@17.0.7/UnixFileAttributes.java:72)
at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(java.base@17.0.7/UnixFileAttributeViews.java:52)
at sun.nio.fs.UnixFileSystemProvider.readAttributes(java.base@17.0.7/UnixFileSystemProvider.java:148)
at sun.nio.fs.LinuxFileSystemProvider.readAttributes(java.base@17.0.7/LinuxFileSystemProvider.java:99)
at java.nio.file.Files.readAttributes(java.base@17.0.7/Files.java:1851)
at java.nio.file.FileTreeWalker.getAttributes(java.base@17.0.7/FileTreeWalker.java:220)
at java.nio.file.FileTreeWalker.visit(java.base@17.0.7/FileTreeWalker.java:277)
at java.nio.file.FileTreeWalker.next(java.base@17.0.7/FileTreeWalker.java:374)
at java.nio.file.Files.walkFileTree(java.base@17.0.7/Files.java:2845)
at com.semmle.frontend.packs.DiskFileSystem.directChildren(DiskFileSystem.java:56)
at com.semmle.frontend.packs.glob.PathMatchTreeWalker.pushStackEntry(PathMatchTreeWalker.java:345)
at com.semmle.frontend.packs.glob.PathMatchTreeWalker.matchDirectory(PathMatchTreeWalker.java:205)
at com.semmle.frontend.packs.glob.PathMatchTreeWalker.walk(PathMatchTreeWalker.java:121)
at com.semmle.frontend.packs.glob.PathMatcher.walkFileTree(PathMatcher.java:270)
at com.semmle.frontend.packs.ScanByManifests.scanTree(ScanByManifests.java:354)
at com.semmle.frontend.packs.ScanByManifests.scanForPacksAndWorkspaces(ScanByManifests.java:154)
at com.semmle.frontend.packs.QlPackResolver.scanForPacksAndWorkspaces(QlPackResolver.java:107)
at com.semmle.frontend.packs.QlPackContainer.computePacksAndWorkspaces(QlPackContainer.java:119)
at com.semmle.frontend.packs.QlPackContainer.lambda$new$0(QlPackContainer.java:38)
at com.semmle.frontend.packs.QlPackContainer$$Lambda$145/0x0000000800d8dd70.apply(Unknown Source)
at com.semmle.api.compilation.CachedOperation.get(CachedOperation.java:91)
- locked <0x0000000591e21578> (a java.lang.Object)
at com.semmle.frontend.packs.QlPackContainer.findPacksByName(QlPackContainer.java:69)
at com.semmle.frontend.packs.QlPackSearchContext.findPacksByName(QlPackSearchContext.java:698)
at com.semmle.frontend.packs.QlPackSearchContext.findPackByName(QlPackSearchContext.java:604)
at com.semmle.frontend.packs.QlPackSearchContext.findPackByVersion(QlPackSearchContext.java:414)
at com.semmle.frontend.packs.QlPackSearchContext.lambda$findAllPackDependencies$19(QlPackSearchContext.java:509)
at com.semmle.frontend.packs.QlPackSearchContext$$Lambda$248/0x0000000800db9560.get(Unknown Source)
at java.util.Optional.orElseGet(java.base@17.0.7/Optional.java:364)
at com.semmle.frontend.packs.QlPackSearchContext.lambda$findAllPackDependencies$20(QlPackSearchContext.java:495)
at com.semmle.frontend.packs.QlPackSearchContext$$Lambda$246/0x0000000800db90d0.apply(Unknown Source)
at java.util.Optional.map(java.base@17.0.7/Optional.java:260)
at com.semmle.frontend.packs.QlPackSearchContext.findAllPackDependencies(QlPackSearchContext.java:467)
at com.semmle.cli2.resolve.ResolveExtensionsCommand.resolvePackDependencies(ResolveExtensionsCommand.java:267)
at com.semmle.cli2.resolve.ResolveExtensionsCommand.executeJSON(ResolveExtensionsCommand.java:70)
at com.semmle.cli2.resolve.ResolveExtensionsCommand.executeJSON(ResolveExtensionsCommand.java:43)
at com.semmle.cli2.picocli.SimpleJsonSubcommand.executeInsistingOnJSONWithExistingMessages(SimpleJsonSubcommand.java:130)
at com.semmle.cli2.picocli.SimpleJsonSubcommand$$Lambda$66/0x0000000800cf0a88.apply(Unknown Source)
at com.semmle.cli2.picocli.SubcommandCommon.withCompilationMessages(SubcommandCommon.java:412)
at com.semmle.cli2.picocli.SimpleJsonSubcommand.executeInsistingOnJSON(SimpleJsonSubcommand.java:119)
at com.semmle.cli2.picocli.PlumbingRunner.call(PlumbingRunner.java:53)
at com.semmle.cli2.picocli.SubcommandCommon.callPlumbingInProcess(SubcommandCommon.java:160)
at com.semmle.cli2.test.CodeqlTestDir$ExtensionValuesCache.calculateExtensions(CodeqlTestDir.java:63)
at com.semmle.cli2.test.CodeqlTestDir$ExtensionValuesCache.get(CodeqlTestDir.java:53)
- locked <0x0000000591e21a18> (a java.util.concurrent.ConcurrentHashMap)
at com.semmle.cli2.test.CodeqlTestDir.resolveExtensions(CodeqlTestDir.java:199)
at com.semmle.cli2.test.CodeqlTestDir$$Lambda$118/0x0000000800d83990.run(Unknown Source)
at java.util.concurrent.CompletableFuture$UniRun.tryFire(java.base@17.0.7/CompletableFuture.java:787)
at java.util.concurrent.CompletableFuture.postComplete(java.base@17.0.7/CompletableFuture.java:510)
at java.util.concurrent.CompletableFuture$AsyncRun.run(java.base@17.0.7/CompletableFuture.java:1810)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.7/ThreadPoolExecutor.java:1136)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.7/ThreadPoolExecutor.java:635)
at java.lang.Thread.run(java.base@17.0.7/Thread.java:833)
```
The root cause seems to be that method `com.semmle.cli2.resolve.ResolveExtensionsCommand.resolvePackDependencies` calls `com.semmle.frontend.packs.QlPackSearchContext.findAllPackDependencies` with `useLegacyResolution` set to true, which ends up conducting a search of the parent of the `codeql` directory.
Please could you either change this behaviour, or make the documentation clearer about where the extraction root for `codeql.zip` should be to avoid this issue?
Hướng dẫn đóng góp
Hướng nghiên cứu
Bắt đầu với trang “Getting started with the CodeQL CLI” được liên kết và xem lại các phần “Extract the zip archive” và “Launch codeql”. Xác định liệu hướng giải quyết dự kiến là hướng dẫn giải nén rõ ràng hơn hay thay đổi hành vi của CLI; tài liệu phải ngăn việc giải nén bên dưới các cây thư mục lớn, đồng thời giữ cho hướng dẫn khởi chạy chính xác.
Do mô hình lập chỉ mục viết ra từ nội dung của issue.
Đánh giá
- Lĩnh vực
- cli, documentation
- Loại issue
- Lỗi
- Độ khó
- 3/5
- Thời gian dự kiến
- 1-2 ngày
- Mức độ hoạt động
- Đình trệ
- Độ rõ ràng
- Khá rõ ràng
- Mức phù hợp với người mới
- 42/100