microsoft / microsoft/TypeScript

⚡ Performance: Project service doesn't cache all fs.statSync

Abierto
#59,338 18 comentarios 0 reacciones 1 asignado Ver en GitHub

@sheetalkamat ya está trabajando en esto.

Desde el 18/7/2024.

Lenguaje dominante
Go
Estrellas
111k
Forks
14.4k
Merge medio
1 d 19 h
PR fusionados (30 d)
117

Descripción

Acknowledgement
  • I acknowledge that issues using this template may be closed without further explanation at the maintainer's discretion.
Comment
🔎 Search Terms

project service fs stat statSync

🙁 Actual behavior

When using typescript-eslint's parserOptions.projectService, type checking APIs switch from the traditional manual TypeScript ts.Program approach to the editor-style ts.ProjectService. We're observing excess calls to the ts.sys.statSync function on some paths in node_modules/ - up to a few dozen for some paths (!).

🙂 Expected behavior

There should be no uncached statSync calls, I'd think? Even if in a persistent session, I'd expect them to be debounced in some way.

As a draft, I added a basic caching Map to statSync and ran a before & after comparison with hyperfine. The results showed a ~7-12% improvement in lint time:

Variant Measurement User Time
Baseline 3.112 s ± 0.033 s 4.382
Caching 2.740 s ± 0.030 s 4.032
diff patch to switch to the Caching variant... 
diff --git a/node_modules/typescript/lib/typescript.js b/node_modules/typescript/lib/typescript.js
index 4baad59..44639d5 100644
--- a/node_modules/typescript/lib/typescript.js
+++ b/node_modules/typescript/lib/typescript.js
@@ -8546,9 +8546,15 @@ var sys = (() => {
         }
       }
     };
+    const statCache = new Map();
     return nodeSystem;
     function statSync(path) {
-      return _fs.statSync(path, { throwIfNoEntry: false });
+      if (statCache.has(path)) {
+        return statCache.get(path);
+      }
+      const result = _fs.statSync(path, { throwIfNoEntry: false });
+      statCache.set(path, result);
+      return result;
     }
     function enableCPUProfiler(path, cb) {
       if (activeSession) {
Additional information about the issue

On the typescript-eslint side:

These are the top 10 most common paths called by statSync... 
32 Error: /Users/josh/repos/performance/node_modules/execa/types/arguments
28 Error: /Users/josh/repos/performance/node_modules/execa/types/stdio
28 Error: /Users/josh/repos/performance/node_modules/execa/types/return
26 Error: /Users/josh/repos/performance/node_modules/execa/types
17 Error: /Users/josh/repos/performance/node_modules/execa/types/methods
14 Error: /Users/josh/repos/performance/node_modules/execa/types/subprocess
9 Error: /Users/josh/repos/performance/node_modules/prettier
9 Error: /Users/josh/repos/performance/node_modules/execa/types/arguments/options.d.ts
8 Error: /Users/josh/repos/performance/node_modules/execa/types/transform
7 Error: /Users/josh/repos/performance/node_modules/execa/types/stdio/type.d.ts
Here's an example call stack from the most common one... 
Error
    at statSync (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:8568:19)
    at fileSystemEntryExists (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:8794:22)
    at Object.directoryExists (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:8816:14)
    at _AutoImportProviderProject.directoryExists (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:182483:40)
    at directoryProbablyExists (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:20914:40)
    at tryAddingExtensions (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:44501:29)
    at loadModuleFromFileNoImplicitExtensions (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:44484:10)
    at loadModuleFromFile (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:44460:40)
    at nodeLoadModuleByRelativeName (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:44409:30)
    at tryResolve (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:44365:25)
    at nodeModuleNameResolverWorker (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:44283:14)
    at nodeNextModuleNameResolverWorker (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:44166:10)
    at nodeNextModuleNameResolver (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:44148:10)
    at resolveModuleName (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:43972:18)
    at resolveModuleNameUsingGlobalCache (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:129141:25)
    at Object.resolve (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:129128:45)
    at resolveNamesWithLocalCache (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:129434:29)
    at Object.resolveModuleNameLiterals (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:129523:12)
    at _AutoImportProviderProject.resolveModuleNameLiterals (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:182461:33)
    at resolveModuleNamesWorker (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:124514:20)
    at resolveModuleNamesReusingOldState (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:124600:14)
    at processImportedModules (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:126117:118)
    at findSourceFileWorker (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:125895:7)
    at findSourceFile (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:125746:20)
    at processImportedModules (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:126143:11)
    at findSourceFileWorker (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:125895:7)
    at findSourceFile (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:125746:20)
    at /Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:125695:22
    at getSourceFileFromReferenceWorker (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:125666:26)
    at processSourceFile (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:125693:5)
    at processRootFile (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:125485:5)
    at /Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:124210:41
    at forEach (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:2387:22)
    at createProgram (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:124210:5)
    at synchronizeHostDataWorker (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:148957:15)
    at synchronizeHostData (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:148853:7)
    at Object.getProgram (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:149029:5)
    at _AutoImportProviderProject.updateGraphWorker (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:183153:41)
    at _AutoImportProviderProject.updateGraph (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:183009:32)
    at _AutoImportProviderProject.updateGraph (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:184155:37)
    at updateProjectIfDirty (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:184785:36)
    at ConfiguredProject2.getPackageJsonAutoImportProvider (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:183741:9)
    at ConfiguredProject2.updateGraph (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:183033:12)
    at ConfiguredProject2.updateGraph (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:184311:24)
    at updateWithTriggerFile (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:184794:11)
    at _ProjectService.reloadConfiguredProject (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:186410:5)
    at ConfiguredProject2.updateGraph (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:184307:29)
    at updateWithTriggerFile (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:184794:11)
    at updateConfiguredProject (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:184802:9)
    at _ProjectService.findCreateOrReloadConfiguredProject (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:187271:44)
    at _ProjectService.tryFindDefaultConfiguredProjectForOpenScriptInfo (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:187295:25)
    at _ProjectService.tryFindDefaultConfiguredProjectAndLoadAncestorsForOpenScriptInfo (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:187341:25)
    at _ProjectService.assignProjectToOpenedScriptInfo (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:187226:27)
    at _ProjectService.openClientFileWithNormalizedPath (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:187421:48)
    at _ProjectService.openClientFile (/Users/josh/repos/performance/node_modules/typescript/lib/typescript.js:187129:17)
    at useProgramFromProjectService (/Users/josh/repos/performance/node_modules/@typescript-eslint/typescript-estree/dist/useProgramFromProjectService.js:61:28)
    at getProgramAndAST (/Users/josh/repos/performance/node_modules/@typescript-eslint/typescript-estree/dist/parser.js:44:100)
    at parseAndGenerateServices (/Users/josh/repos/performance/node_modules/@typescript-eslint/typescript-estree/dist/parser.js:155:11)
    at Object.parseForESLint (/Users/josh/repos/performance/node_modules/@typescript-eslint/parser/dist/parser.js:101:80)
    at Object.parse (/Users/josh/repos/performance/node_modules/eslint/lib/languages/js/index.js:186:26)
    at parse (/Users/josh/repos/performance/node_modules/eslint/lib/linter/linter.js:931:29)
    at Linter._verifyWithFlatConfigArrayAndWithoutProcessors (/Users/josh/repos/performance/node_modules/eslint/lib/linter/linter.js:1696:33)
    at Linter._verifyWithFlatConfigArray (/Users/josh/repos/performance/node_modules/eslint/lib/linter/linter.js:2062:21)
    at Linter.verify (/Users/josh/repos/performance/node_modules/eslint/lib/linter/linter.js:1528:61)
    at Linter.verifyAndFix (/Users/josh/repos/performance/node_modules/eslint/lib/linter/linter.js:2299:29)
    at verifyText (/Users/josh/repos/performance/node_modules/eslint/lib/eslint/eslint.js:498:48)
    at /Users/josh/repos/performance/node_modules/eslint/lib/eslint/eslint.js:939:40
    at async Promise.all (index 1)
    at async ESLint.lintFiles (/Users/josh/repos/performance/node_modules/eslint/lib/eslint/eslint.js:880:25)
    at async Object.execute (/Users/josh/repos/performance/node_modules/eslint/lib/cli.js:521:23)
    at async main (/Users/josh/repos/performance/node_modules/eslint/bin/eslint.js:153:22)

cc @sheetalkamat as FYI, after a pairing with @jakebailey.

Guía de contribución

Abrir la guía de contribución

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.