mhx / mhx/dwarfs

mkdwarfs stuck on scanning when using NTFS on Linux

Open
#380 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

stalled
Dominant language
C++
Stars
2.6k
Forks
92
PR merge metrics
No merged PRs in 30d

Description

Hi, I've run into a slight problem while using DwarFS, or more specifically, mkdwarfs. I primarily run Linux, and my Linux partition uses Btrfs. When I use mkdwarfs on directories in my Linux partition, everything works as expected. However, I have a Windows partition and several hard drives formatted in NTFS. When I use mkdwarfs in one of these partitions or drives, mkdwarfs runs until it starts scanning, then freezes indefinitely. Below is an example with logging set at verbose, for reference.

```
◄ 0s ◎ mkdwarfs -i './Library/' -o './library.dwarfs' --log-level=verbose □ user/F42A36B32A3672AA 14:03
V 14:04:24.834004 [mkdwarfs_main.cpp:1375] using memory limit of 751.4 MiB
V 14:04:24.834119 [mkdwarfs_main.cpp:1410] [--order]
V ............... default: nilsimsa:max_children=16384:max_cluster_size=16384
V 14:04:24.834138 [mkdwarfs_main.cpp:1420] [--max-lookback-blocks]
V ............... default: 1
V 14:04:24.834144 [mkdwarfs_main.cpp:1430] [--window-size]
V ............... default: 12
V 14:04:24.834148 [mkdwarfs_main.cpp:1440] [--window-step]
V ............... default: 3
V 14:04:24.834152 [mkdwarfs_main.cpp:1450] [--bloom-filter-size]
V ............... default: 4
V 14:04:24.834556 [mkdwarfs_main.cpp:1499] [--compression]
V ............... default: zstd [level=22]
I 14:04:24.834867 [scanner.cpp:663] scanning "/run/media/user/F42A36B32A3672AA/Library"
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯

7 dirs, 0/0 soft/hard links, 2/6 files, 0 other
original size: 4.074 MiB, hashed: 0 B (0 files, 0 B/s)
scanned: 520 KiB (2 files, 0 B/s), categorizing: 0 B/s
saved by deduplication: 0 B (0 files), saved by segmenting: 0 B
filesystem: 0 B in 0 blocks (0 chunks, 2 fragments, 2/6 inodes)
compressed filesystem: 0 blocks/0 B written, using 43.23 MiB of RAM
```
While this is going on, the output file is created (library.dwarfs, in my example) as a file 0 bytes in size. Also, the CPU usage from mkdwarfs continues despite the lack of progress until I cancel the command.

If I use mkdwarfs on Windows, this issue doesn't occur: mkdwarfs works on files in the Windows partition and my NTFS formatted drives, presumably because NTFS is native to Windows. I've not extensively tested other formats on Linux, but at least exFAT seems to work without issues.

Would there be any way for me to get mkdwarfs to work with NTFS drives on Linux? Thank you!

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the hang with mkdwarfs on an NTFS-mounted path using the verbose command and compare it with Btrfs or exFAT. Start at the scanning path shown in scanner.cpp:663 and review the related mkdwarfs_main.cpp logging; done means NTFS input completes scanning or reports a bounded, actionable error instead of consuming CPU indefinitely.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, linux
Domain
cli, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.