Statement of the problem and review of possible existing solutions

# High level overview

An increasing number of projects need to operate on growing large collections of files. Having (hundreds of ) thousands of files within a file tree is no longer atypical e.g. in neuroimaging. Many supplementary tools/functions need to "walk" the file tree and return result of some function. Such result should not change between invocations if nothing in the file tree changes (no file added/removed/changed) and theoretically output of the function should remain the same.

Situation scales down all the way to individual files: function could operate on a specific file path and return result which should not change if file does not change.

# "Fulfilled" use-case

- `git` already does some level of caching to speed up `git status` (see comment in datalad below), so I think it should be possible to provide some generic helper which would cache the result of operation on some path argument (file or a tree).
- `dandi-cli`: loading metadata from .nwb file which takes a while. For that purpose, based on joblib.Memory I came up with a [PersistentCache](https://github.com/dandi/dandi-cli/blob/master/dandi/support/cache.py#L14) class to provide a general persistent (across processes) cache, and a `.memoize_path` method which is a decorator to decorate any function which takes path to a file as the first argument.

# Implementation

- Dunno yet what to do given a file tree, and either there is filesystem specifics involved to be (ab)used. Probably should not rely on active monitoring (e.g. inotify) to not depend on some daemon to be running.
- That `PersistentCache` mentioned above also can account (see [example](https://github.com/dandi/dandi-cli/blob/master/dandi/pynwb_utils.py#L27)) for versions of the relevant python modules placing them into signature, so if there is an update, cached result will not be used.
- probably should accept a list of paths changes to which could be safely ignored
- consider support to account for hierarchical organization of files/directories -- so it would be the caching helper which would cause re-invocation only on relevant sb-trees while reloading cached for the others which had not changed

# Target use-cases

- DataLad: `datalad status` and possibly some (e.g. `diff`) other commands operation. Whenever `git status` already uses some smart caching of the results, so subsequent invocation takes advantage of it, there is nothing like that within datalad yet.

- PyBIDS: construction of the `BIDSLayout` could take awhile. If instance (or at least a list of walked paths) could be cached -- would speed up subsequent invocations. (ref: https://github.com/bids-standard/pybids/issues/609#issuecomment-627492261)

- `dandi-cli`: ATM we cache per file, but as larger in number of files datasets appear, we might like to cache results on full file trees

If anyone (attn @kyleam @mih @vsoch @con @tyarkoni @effigies @satra) knows an already existing solution -- would be great!

Start with the issue's proposed use cases and review the referenced dandi-cli files, dandi/support/cache.py and dandi/pynwb_utils.py, alongside DataLad status and PyBIDS BIDSLayout. Investigate existing caching solutions and define requirements for files, trees, ignored paths, module versions, and unchanged results. Done requires a decided scope and implementation direction, but no repository entry point or test is named.

Written by the indexing model from the issue text.