apache / apache/arrow

[C++] Improve arrow::fs::FileSelect performance for `IsFile()` and `IsDirectory()`

Open
#33,618 12 comments 0 reactions 1 assignee Claimed by @anjakefala View on GitHub
Component: C++ Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the enhancement requested

Here is an example `std::filesystem`:

```
for (const auto &path : std::filesystem::directory_iterator(dataDir)) {
if (path.is_regular_file()) {
// logic here
}
}
```

Here is a similar example using `arrow::fs::FileSelect()`:

```
auto selector = arrow::fs::FileSelector();
selector.base_dir = dir;
selector.allow_not_found = false;
selector.recursive = false;
auto fileInfo = fs_->GetFileInfo(selector);
for (const auto &pathInfo: fileInfo.ValueOrDie()) {
pathInfo.IsFile();
// OR
pathInfo.IsDirectory();
}
```

The `arrow::fs::FileSelector()` example takes 10x as long. Under the hood there is a stat() call which fills in extra metadata information, and this increases time spent making the call. `std::filesystem` instead uses `readdir`, which checks whether an entry is a file or directory, but does not provide extra metadata information. This results in a much faster system call.

`std::filesystem::directory_iterator` gives the user a `directory_entry` which does have a `file_size` attribute. Calling `file_size` the first time triggers a stat call to the file but a stat call is not made if you don't call `file_size`.

I propose a similar adjustment to the `arrow::fs` API, which allows a user to perform an entry identity check with a `readdir` system call, such that a `stat` call is only used if the user requests the extra metadata (e.g. file size and mtime).

### Component(s)

C++

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.