Discussion: metro-file-map roadmap ideas that would be fantastic for monorepos

TL;DR: thank you for adding symlink support! It works extremely well for us, but there are some directions we think metro-file-map could be taken that would work better for large non-Facebook monorepos (non-Facebook as in, on standard filesystems).

These directions include (more below):

• Lazily computing content.sha1hex for files in node_modules
• Lazily loading files in node_modules into TreeFS (rather than eagerly loading all node_modules)
• Tolerating node_modules not being returned in the Watchman query

---

Our usecase

At Discord, we have a large monorepo, whose node_modules are managed by pnpm, with the following structure:

/
  node_modules/
    .pnpm/ (full of highly specific directories containing actual third party code)

  discord_app/
    node_modules/ (full of symlinks to ../../node_modules/.pnpm/* and other parts of the repo)

  discord_common/js/packages/... (full of other code used by the app)

Because of this layout (which we can't really change), Metro ends up issuing a Watchman query that can take around 20 seconds on even the fastest machines: it requests the content.sha1hex of every single file in node_modules, which for us, is around 300k.

If we remove node_modules as a watchFolder, then no node_modules end up in the file map, and Metro cannot find them (which is understandable).

Because of this behavior of metro-file-map, we're unable to exclude node_modules/.pnpm from our Watchman config, which means that watchman watch-project . is also incredibly slow—even though Metro is the only tool we use that actually needs to have Watchman read node_modules/.pnpm (the other tools just follow symlinks).

Some directions I've explored

I set out this past week to try patching Metro in various ways and seeing how behaviors improved. I have a couple of approaches that I think could be worth upstreaming:

Idea 1: Lazily computing sha1s

I noticed that the Watchman query issued by Metro was taking around 20 seconds, and realized that it went down to <1s when removing content.sha1hex as a field from the query. So I tried the following:

diff --git a/packages/metro-file-map/src/lib/TreeFS.js b/packages/metro-file-map/src/lib/TreeFS.js
index 28971ff3..1038cabb 100644
--- a/packages/metro-file-map/src/lib/TreeFS.js
+++ b/packages/metro-file-map/src/lib/TreeFS.js
@@ -22,6 +22,8 @@ import H from '../constants';
 import {RootPathUtils} from './RootPathUtils';
 import invariant from 'invariant';
 import path from 'path';
+import crypto from 'crypto';
+import fs from 'fs';
 
 type DirectoryNode = Map<string, MixedNode>;
 type FileNode = FileMetaData;
@@ -185,7 +187,15 @@ export default class TreeFS implements MutableFileSystem {
 
   getSha1(mixedPath: Path): ?string {
     const fileMetadata = this._getFileData(mixedPath);
-    return (fileMetadata && fileMetadata[H.SHA1]) ?? null;
+    if (fileMetadata == null) return null;
+
+    let sha1 = fileMetadata[H.SHA1] ?? null;
+    if (sha1 == null) {
+      const content = fs.readFileSync(mixedPath);
+      sha1 = crypto.createHash('sha1').update(content).digest('hex');
+      fileMetadata[H.SHA1] = sha1;
+    }
+    return sha1;
   }
 
   exists(mixedPath: Path): boolean {
diff --git a/packages/metro/src/node-haste/DependencyGraph/createFileMap.js b/packages/metro/src/node-haste/DependencyGraph/createFileMap.js
index 1533b941..37f193ba 100644
--- a/packages/metro/src/node-haste/DependencyGraph/createFileMap.js
+++ b/packages/metro/src/node-haste/DependencyGraph/createFileMap.js
@@ -80,7 +80,7 @@ function createFileMap(
         })),
     perfLoggerFactory: config.unstable_perfLoggerFactory,
     computeDependencies,
-    computeSha1: true,
+    computeSha1: false,
     dependencyExtractor: config.resolver.dependencyExtractor,
     enableHastePackages: config?.resolver.enableGlobalPackages,
     enableSymlinks: config.resolver.unstable_enableSymlinks,

Disgusting. But it made it so that we never get stuck on a slow query. Now, I think that asking watchman for sha1's is actually really reasonable—for app code. For node_modules, we're asking it to sha1sum hundreds of thousands of files that Metro will likely never actually have to read.

One thing to consider may be a hybrid approach here, then: issue two watchman queries—one for all the app code (with content.sha1hex), and one for all node_modules (without content.sha1hex). Then, around the moment the sha1 is needed by the transformer, we compute it and write it to the filemap (preferably more elegantly than above).

However, this still has a problem: we are still loading hundreds of thousands of files into the filemap, most of which we'll probably never read, which is probably part of why, at Discord, we have to give Metro 16 gigabytes of RAM.

Idea 2: Excluding node_modules from the query entirely

This would, imo, be the most monorepo-friendly approach. Instead of querying watchman for node_modules, Metro could resolve imports as they are encountered (by, yes, following symlinks), and just read from the filesystem (caching as you go). I understand this would probably be a large behavioral change to metro-file-map (much larger than Idea 1).

But the downstream benefits of this would be massive:

1. Folks would be able to exclude node_modules/.pnpm in .watchmanconfig, which for us at least, takes watch-project down from 500k files to just 200k, speeding it up by the same proportion.
2. It would prevent Watchman from recrawling whenever node_modules changes (because the most frequently-changing parts would be ignored).
3. It would drastically reduce metro-file-map's memory footprint for monorepos with many node_modules.
4. It would reduce the memory footprint of Watchman's cache.

---

In all this exploration, I've started to feel a little bit comfortable inside of the Metro monorepo, and I'd be interested in contributing some of these ideas—but wanted to talk direction/interest first. I hope this perspective from someone dealing with a hefty repo has been valuable. Thanks :)

The discussion points to packages/metro-file-map/src/lib/TreeFS.js, especially getSha1, and packages/metro/src/node-haste/DependencyGraph/createFileMap.js, where computeSha1 is configured. Read those entry points and the Watchman query behavior first; the issue needs an agreed direction and concrete acceptance criteria before completion can be defined.