mkdwarfs stuck on scanning when using NTFS on Linux
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 2.6k
- Forks
- 92
- PR merge metrics
- No merged PRs in 30d
Description
Hi, I've run into a slight problem while using DwarFS, or more specifically, mkdwarfs. I primarily run Linux, and my Linux partition uses Btrfs. When I use mkdwarfs on directories in my Linux partition, everything works as expected. However, I have a Windows partition and several hard drives formatted in NTFS. When I use mkdwarfs in one of these partitions or drives, mkdwarfs runs until it starts scanning, then freezes indefinitely. Below is an example with logging set at verbose, for reference.
```
◄ 0s ◎ mkdwarfs -i './Library/' -o './library.dwarfs' --log-level=verbose □ user/F42A36B32A3672AA 14:03
V 14:04:24.834004 [mkdwarfs_main.cpp:1375] using memory limit of 751.4 MiB
V 14:04:24.834119 [mkdwarfs_main.cpp:1410] [--order]
V ............... default: nilsimsa:max_children=16384:max_cluster_size=16384
V 14:04:24.834138 [mkdwarfs_main.cpp:1420] [--max-lookback-blocks]
V ............... default: 1
V 14:04:24.834144 [mkdwarfs_main.cpp:1430] [--window-size]
V ............... default: 12
V 14:04:24.834148 [mkdwarfs_main.cpp:1440] [--window-step]
V ............... default: 3
V 14:04:24.834152 [mkdwarfs_main.cpp:1450] [--bloom-filter-size]
V ............... default: 4
V 14:04:24.834556 [mkdwarfs_main.cpp:1499] [--compression]
V ............... default: zstd [level=22]
I 14:04:24.834867 [scanner.cpp:663] scanning "/run/media/user/F42A36B32A3672AA/Library"
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
7 dirs, 0/0 soft/hard links, 2/6 files, 0 other
original size: 4.074 MiB, hashed: 0 B (0 files, 0 B/s)
scanned: 520 KiB (2 files, 0 B/s), categorizing: 0 B/s
saved by deduplication: 0 B (0 files), saved by segmenting: 0 B
filesystem: 0 B in 0 blocks (0 chunks, 2 fragments, 2/6 inodes)
compressed filesystem: 0 blocks/0 B written, using 43.23 MiB of RAM
```
While this is going on, the output file is created (library.dwarfs, in my example) as a file 0 bytes in size. Also, the CPU usage from mkdwarfs continues despite the lack of progress until I cancel the command.
If I use mkdwarfs on Windows, this issue doesn't occur: mkdwarfs works on files in the Windows partition and my NTFS formatted drives, presumably because NTFS is native to Windows. I've not extensively tested other formats on Linux, but at least exFAT seems to work without issues.
Would there be any way for me to get mkdwarfs to work with NTFS drives on Linux? Thank you!
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the hang with mkdwarfs on an NTFS-mounted path using the verbose command and compare it with Btrfs or exFAT. Start at the scanning path shown in scanner.cpp:663 and review the related mkdwarfs_main.cpp logging; done means NTFS input completes scanning or reports a bounded, actionable error instead of consuming CPU indefinitely.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, linux
- Domain
- cli, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100