fish-shell / fish-shell/fish-shell

`**` glob witnessing dataless directories materializes them on macOS

Open
#8,399 21 comments 0 reactions 1 assignee Claimed by @zanchey View on GitHub
enhancement macos
Dominant language
Rust
Stars
34.2k
Forks
2.4k
Avg merge
2d 8h
Merged PRs (30d)
13

Description

Files that exist "dataless" on disk are being forced to download and sync locally just by iterating over them with `**`.

This affects stuff like iCloud paths, likely dropbox, Google Drive, box accounts etc. (assuming they now use this) and definitely S3/DAV/FTP/whatever shares through [Strongsync](https://www.expandrive.com/strongsync/) which uses the [FileProvider API](https://developer.apple.com/documentation/fileprovider).

"dataless" is a state supported by APFS in which a file or directory is a placeholder, and its children (for a directory) or content (for a file) will be fetched when the user tries to read it.

`ls -l%` is an easy way to see if files are dataless or not.

```
-% Distinguish dataless files and directories with a '%' character in long (-l) output, and
don't materialize dataless directories when listing them.
```

I first noticed this when trying to determine why `**` in my home directory could take minutes. After attempting to debug it for a while I noticed all of a sudden it was quite fast. ~~But my hard disk was almost empty.~~

```fish
~/L/C/Strongsync-SPMDrive►echo *
bugreceipt.png Inspection Entry.webloc Inspection Record.webloc log_0D1CD0_1-21.webloc OrgData-smallpartsmfg.com-1p2uljz3ras55u-20160720.csv.webloc Shared Drives Shared with Me sonicwall-TZ_400-6_2_3_1-19n.exp SPM techSupport_0D1CD0_1-21.wri test.pdf textual.mov Untitled spreadsheet.webloc
~/L/C/Strongsync-SPMDrive►ls -l%
total 0
-rw-------% 1 floam staff 317 May 6 2015 Inspection Entry.webloc
-rw-------% 1 floam staff 324 Oct 29 2014 Inspection Record.webloc
-rw-------% 1 floam staff 324 Oct 6 2016 OrgData-smallpartsmfg.com-1p2uljz3ras55u-20160720.csv.webloc
drwx------% 20 floam staff 640 Sep 29 2015 SPM/
drwx------% 2 floam staff 64 Nov 2 00:13 Shared Drives/
drwx------% 10 floam staff 320 Nov 2 00:13 Shared with Me/
-rw-------% 1 floam staff 324 Oct 29 2014 Untitled spreadsheet.webloc
-rw-------% 1 floam staff 78132 Apr 23 2016 bugreceipt.png
-rw-------% 1 floam staff 324 Jan 21 2016 log_0D1CD0_1-21.webloc
-rw-------% 1 floam staff 808062 Jan 21 2016 sonicwall-TZ_400-6_2_3_1-19n.exp
-rw-------% 1 floam staff 738966 Jan 21 2016 techSupport_0D1CD0_1-21.wri
-rw-------% 1 floam staff 492607 Apr 1 2016 test.pdf
-rw-------% 1 floam staff 8467525 Apr 18 2016 textual.mov
~/L/C/Strongsync-SPMDrive►echo **
bugreceipt.png Inspection Entry.webloc Inspection Record.webloc log_0D1CD0_1-21.webloc OrgData-smallpartsmfg.com-1p2uljz3ras55u-20160720.csv.webloc Shared Drives Shared with Me Shared with Me/BOTTOM.step Shared with Me/image0.jpeg Shared with Me/image1.jpeg Shared with Me/image2.jpeg Shared with Me/image3.jpeg Shared with Me/image5.png Shared with Me/sw_tz-400_eng_6.2.5.0_6.2.5_15n_858612.sig Shared with Me/TOP.step sonicwall-TZ_400-6_2_3_1-19n.exp SPM SPM/IMG_0393.JPG SPM/IMG_0394.JPG SPM/IMG_0395.JPG SPM/IMG_0396.JPG SPM/IMG_0397.JPG SPM/IMG_0398.JPG SPM/IMG_0399.JPG SPM/IMG_0400.JPG SPM/IMG_0401.JPG SPM/IMG_0402.JPG SPM/IMG_0403.JPG SPM/IMG_0404.JPG SPM/IMG_0405.JPG SPM/IMG_0406.JPG SPM/IMG_0407.JPG SPM/IMG_0408.JPG SPM/IMG_0412.JPG SPM/IMG_0413.JPG techSupport_0D1CD0_1-21.wri test.pdf textual.mov Untitled spreadsheet.webloc
~/L/C/Strongsync-SPMDrive►echo $CMD_DURATION
9486
ls -l%
total 0
-rw-------% 1 floam staff 317 May 6 2015 Inspection Entry.webloc
-rw-------% 1 floam staff 324 Oct 29 2014 Inspection Record.webloc
-rw-------% 1 floam staff 324 Oct 6 2016 OrgData-smallpartsmfg.com-1p2uljz3ras55u-20160720.csv.webloc
drwx------@ 20 floam staff 640 Sep 29 2015 SPM/
drwx------@ 2 floam staff 64 Nov 2 00:13 Shared Drives/
drwx------@ 10 floam staff 320 Nov 2 00:13 Shared with Me/
-rw-------% 1 floam staff 324 Oct 29 2014 Untitled spreadsheet.webloc
-rw-------% 1 floam staff 78132 Apr 23 2016 bugreceipt.png
-rw-------% 1 floam staff 324 Jan 21 2016 log_0D1CD0_1-21.webloc
-rw-------% 1 floam staff 808062 Jan 21 2016 sonicwall-TZ_400-6_2_3_1-19n.exp
-rw-------% 1 floam staff 738966 Jan 21 2016 techSupport_0D1CD0_1-21.wri
-rw-------% 1 floam staff 492607 Apr 1 2016 test.pdf
-rw-------% 1 floam staff 8467525 Apr 18 2016 textual.mov
```

Notice the three directories that have apparently forced remote file servers to be queried, etc.

Neither `ls` (except -l sans -%, see quoted manual above), `du`, nor `echo **` on zsh has this behavior. We should figure out how to avoid it.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.