[C++] Improve arrow::fs::FileSelect performance for `IsFile()` and `IsDirectory()`
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the enhancement requested
Here is an example `std::filesystem`:
```
for (const auto &path : std::filesystem::directory_iterator(dataDir)) {
if (path.is_regular_file()) {
// logic here
}
}
```
Here is a similar example using `arrow::fs::FileSelect()`:
```
auto selector = arrow::fs::FileSelector();
selector.base_dir = dir;
selector.allow_not_found = false;
selector.recursive = false;
auto fileInfo = fs_->GetFileInfo(selector);
for (const auto &pathInfo: fileInfo.ValueOrDie()) {
pathInfo.IsFile();
// OR
pathInfo.IsDirectory();
}
```
The `arrow::fs::FileSelector()` example takes 10x as long. Under the hood there is a stat() call which fills in extra metadata information, and this increases time spent making the call. `std::filesystem` instead uses `readdir`, which checks whether an entry is a file or directory, but does not provide extra metadata information. This results in a much faster system call.
`std::filesystem::directory_iterator` gives the user a `directory_entry` which does have a `file_size` attribute. Calling `file_size` the first time triggers a stat call to the file but a stat call is not made if you don't call `file_size`.
I propose a similar adjustment to the `arrow::fs` API, which allows a user to perform an entry identity check with a `readdir` system call, such that a `stat` call is only used if the user requests the extra metadata (e.g. file size and mtime).
### Component(s)
C++
Contributor guide
Assessment
This issue has not been assessed yet.