Add symlink following to watchman query engine
- Dominant language
- C++
- Stars
- 13.7k
- Forks
- 1.1k
- PR merge metrics
- No merged PRs in 30d
Description
Watchman does not currently "support symlinks".
https://github.com/facebook/watchman/issues/105#issuecomment-206463325 has some thoughts on how we might address this.
This particular issue encompasses one portion of this effort:
# Following symlinks while evaluating queries
The query evaluator in watchman works by iterating over a set of nodes provided by a generator and filtering by a user-provided expression tree.
The critical portion of this executes while holding a lock on the data associated with the watched tree (we call this a "root").
# To add symlink following to the engine:
- Add a `{"follow_symlinks": true}` field to the query specification. It must default to false for backwards compatibility. Look in query/parse.c for the query parsing code.
- When executing, if a node is a symlink and `follow_symlinks` is true, the engine should examine the `symlink_target` stored in the `struct watchman_file` and resolve it to the containing root in a similar fashion to the way that watchman watch-project functions (look at #347 for how this is done at symlink discovery time). This runtime evaluation should probably not use `realpath` to resolve the symlink from the filesystem; with the auto-watching from #347 we should be able to resolve the symlink using our in-memory state contained in the roots and the appropriate `watchman_files` structs. Once resolved, the engine should resolve the `struct watchman_file` from that target project.
- The newly resolved target may itself be a symlink; we should allow some configurable number of iterations of target resolution before giving up on the resolution, perhaps by allowing `follow_symlinks` to be an integer specifying the limit? `{"follow_symlinks": 32}` with `true` meaning use a default reasonable limit.
- If successfully resolved, the resolved file pointer should be taken and used as the source of data for the expression engine.
- If not resolvable, we can treat that entry as not matching the expression term and continue to the next term.
# Complications:
- If the target of the symlink is a file from another root, we cannot safely examine its struct `watchman_file` without obtaining a lock on that root. We need to be careful about deadlock avoidance. We recently introduced read locks for queries that make this slightly less of a concern, but the possibility still exists in some cases. I'd be fine with limiting the use of `follow_symlinks` to the read-lock only query path. In practice this is for any query except those that use the server side named cursor feature (`n:foo` from https://facebook.github.io/watchman/docs/clockspec.html). We can detect this in the code in `query/parse.c` and error out during query parsing if `follow_symlinks` is enabled together with the `since` generator and a named clock is used.
- The symlink target resolution must not change the name that is visible to the name matching expression terms. For example, if I have the symlink A that points to file B, if I ask watchman to filter files named A, we must not treat the name of A as B just because of the symlink resolution. Look at the `wholename` member of `struct w_query_ctx` and the `w_query_ctx_get_wholename()` function.
# Path and Glob Generator
These two walking strategies need some special support, as they both traverse the in-memory directory structure. When `follow_symlinks` is enabled, they will need to respect that option and then jump to the appropriate portion of the logical tree.
Contributor guide
Assessment
This issue has not been assessed yet.