[Feature] Overlay FS

• I'd be willing to implement this feature
• This feature can already be implemented through a plugin

Describe the user story

Some modules like babel or @angular like to write inside other modules their caching results. I want Yarn 2 to allow them doing so.

Describe the solution you'd like

I want Yarn 2 to implement overlay fs layer, which will be a copy on write fs that stores results of installed modules files modifications done by other modules. For example if module foo writes new files inside .yarn/cache/bar-xyz.zip/node_modules/bar/.cache, the files were really written to .yarn/overlay/bar-xyz.zip/node_modules/bar/.cache, where .yarn/overlay/bar-xyz.zip is a directory, not a zip file. When any module tries to read the files from this destination at any time in the future it should be able to see them. At the same time real zip file: .yarn/cache/bar-xyz.zip should not be modified.

.yarn/overlay should not be committed into VCS.

For efficient decision making whether we should read from overlay or from zip for the given path and to not double the syscalls, we can maintain mem-efficient tree of all the files available inside .yarn/overlay. The key of the tree will be a 128-bit hash of the directory part of the path and the value is the list of all the files inside the directory. This way we will have compact path representation inside the tree and at the same time we will be able to recursively remove directories from the tree.

Describe the drawbacks of your solution

The drawback is an implementation of another not very trivial mechanism. Another drawback is increased mem usage is needed, hopefully not too big, to avoid doubling of fs syscalls.

Describe alternatives you've considered

The alternative is to forbid writing to .zip files and recommend unplugging modules to which the writes might happen as we do it now, or recommend using other locations for temporary files. This is inconvenient for the tools that already store their cache inside the modules.

The issue names no files, tests, or entry points. Start by locating Yarn's existing zip filesystem and the unplugging behavior mentioned as the alternative, then determine how an overlay would integrate with them. Done means supporting copy-on-write module changes through .yarn/overlay without modifying cached zip files, while keeping the overlay out of version control.