Concurrent disk I/O

Commands that perform a lot of disk I/O (cp, mv, rm, etc.) are likely to complete much faster, especially when operating on SSDs[^1] but potentially also on slower disks[^2], if they issue concurrent I/O requests (ideally asynchronously and in bulk[^3], but synchronously via a thread pool could also work).

Examples of cases where this could be very beneficial include:

• cp -r <source> <dest> where <source> contains large number of files/subdirectories
• cp <sources> <dest> where <sources> is a large number of files
• rm -r <source> where <source> contains large number of files/subdirectories
• rm <sources> where <sources> is a large number of files
• mv <sources> <dest> where <sources> is a large number of files
• ln <sources> <dest> where <sources> is a large number of files
• ls <path> and ls -r <path> where <path> contains large number of files/subdirectories
• du <path> where <path> contains large number of files/subdirectories

(and so on, the list above is not exhaustive: examples include stat, readlink, touch, chown, chgrp, chmod, etc.)

Whether this behavior should be opt-out/default-off or opt-in/default-on, I lean towards opt-out but I must admit I am not sure. With ever-increasing core counts and ever-higher capacity disks, I would argue there is a pretty strong case for the behavior to be opt-out rather than opt-in. At the same time, there may be concerns about these tools suddenly generating large I/O spikes, in which case it should be opt-in. In all cases, the proposed behavior should never affect the correctness of the command, or its semantics (e.g. ordering of results, behavior on error, etc.).

Currently, the closest that can be achieved with standard tools is running processes in parallel by using something like xargs -P<n>, but this is somewhat easy to get wrong (e.g. forgetting to use -0, dynamically choosing between -I{} and normal mode based on how many files we need to operate on and how long their names are, etc) and it does not, in general, maintain the standard semantics of the tools (e.g. ordering needs to be done separately, handling errors is left to the user, etc.)

Update: in this section I will collect any available evidence, concerning performance, for or against this proposal:

For

• My quick benchmarks using find | xargs -P showing 50-100% improvements in wall time
• Fast Unix Commands (fuc) benchmarks showing up to 250% improvements in wall time
• bfs 3.0: the fastest find yet! showing wall times almost 3x shorter than find
• (please ping me so that I can add any further relevant evidence)

Against

• (nothing yet, please ping me so that I can add any relevant evidence)

[^1]: as they tend to deliver higher throughput when operating with high queue depths
[^2]: as the OS can potentially coalesce a greater number of logical writes into a smaller number of disk write operations
[^3]: especially if via io_uring or similar mechanisms that can guarantee that in practice file operations never block, and that support batch submission

Start by examining the listed commands and the existing xargs -P and benchmark evidence linked in the issue. Define which command behaviors can use concurrent I/O while preserving ordering, error handling, and other semantics, then validate the chosen approach against the cited performance cases.