facebook / facebook/watchman

Add symlink following to watchman query engine

Open
#349 0 comments 6 reactions 0 assignees View on GitHub
enhancement
Dominant language
C++
Stars
13.7k
Forks
1.1k
PR merge metrics
No merged PRs in 30d

Description

Watchman does not currently "support symlinks".
https://github.com/facebook/watchman/issues/105#issuecomment-206463325 has some thoughts on how we might address this.

This particular issue encompasses one portion of this effort:
# Following symlinks while evaluating queries

The query evaluator in watchman works by iterating over a set of nodes provided by a generator and filtering by a user-provided expression tree.
The critical portion of this executes while holding a lock on the data associated with the watched tree (we call this a "root").
# To add symlink following to the engine:
- Add a `{"follow_symlinks": true}` field to the query specification. It must default to false for backwards compatibility. Look in query/parse.c for the query parsing code.
- When executing, if a node is a symlink and `follow_symlinks` is true, the engine should examine the `symlink_target` stored in the `struct watchman_file` and resolve it to the containing root in a similar fashion to the way that watchman watch-project functions (look at #347 for how this is done at symlink discovery time). This runtime evaluation should probably not use `realpath` to resolve the symlink from the filesystem; with the auto-watching from #347 we should be able to resolve the symlink using our in-memory state contained in the roots and the appropriate `watchman_files` structs. Once resolved, the engine should resolve the `struct watchman_file` from that target project.
- The newly resolved target may itself be a symlink; we should allow some configurable number of iterations of target resolution before giving up on the resolution, perhaps by allowing `follow_symlinks` to be an integer specifying the limit? `{"follow_symlinks": 32}` with `true` meaning use a default reasonable limit.
- If successfully resolved, the resolved file pointer should be taken and used as the source of data for the expression engine.
- If not resolvable, we can treat that entry as not matching the expression term and continue to the next term.
# Complications:
- If the target of the symlink is a file from another root, we cannot safely examine its struct `watchman_file` without obtaining a lock on that root. We need to be careful about deadlock avoidance. We recently introduced read locks for queries that make this slightly less of a concern, but the possibility still exists in some cases. I'd be fine with limiting the use of `follow_symlinks` to the read-lock only query path. In practice this is for any query except those that use the server side named cursor feature (`n:foo` from https://facebook.github.io/watchman/docs/clockspec.html). We can detect this in the code in `query/parse.c` and error out during query parsing if `follow_symlinks` is enabled together with the `since` generator and a named clock is used.
- The symlink target resolution must not change the name that is visible to the name matching expression terms. For example, if I have the symlink A that points to file B, if I ask watchman to filter files named A, we must not treat the name of A as B just because of the symlink resolution. Look at the `wholename` member of `struct w_query_ctx` and the `w_query_ctx_get_wholename()` function.
# Path and Glob Generator

These two walking strategies need some special support, as they both traverse the in-memory directory structure. When `follow_symlinks` is enabled, they will need to respect that option and then jump to the appropriate portion of the logical tree.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.