apache / apache/arrow

[C++][Dataset] Devise a mechanism to limit the total "system ram" (process + cache) used by dataset writes

Open
#30,179 2 comments 0 reactions 0 assignees View on GitHub
Component: C++ Priority: Critical Status: needs champion Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

The dataset writer now correctly applies backpressure.  However, that backpressure is only applied when the write calls slow down.  This only happens when the OS disk cache fills up.

However, filling up the OS disk cache is undesirable.  It will cause all running processes to get swapped (assuming the system has any swap configured) and will make the system unusable for anything else.

This typically has no actual benefit to the dataset write.  The marginal performance boost provided by the extra RAM is often not worth the cost.

One way to do this would be to use direct I/O (although that comes with a plethora of warnings).  Another way might be to flag the output was WONTNEED but I don't know for sure if this works (the OS might still cache it so that it can satisfy the write call quickly).  Another way might be to somehow track how much disk cache is being used for writes but that would get complex.  I'm sure there are other ways I'm just not aware of yet.

**Reporter**: [Weston Pace](https://issues.apache.org/jira/browse/ARROW-14635) / @westonpace
**Assignee**: [Ziheng Wang](https://issues.apache.org/jira/browse/ARROW-14635) / @marsupialtail
#### Related issues:
- [[Python][C++] O_DIRECT write support ](https://github.com/apache/arrow/issues/32418) (supercedes)
- [[C++][R]Opening a multi-file dataset and writing a re-partitioned version of it fails](https://github.com/apache/arrow/issues/18944) (is depended upon by)
#### PRs and other links:
- [GitHub Pull Request #13640](https://github.com/apache/arrow/pull/13640)
- [GitHub Pull Request #13662](https://github.com/apache/arrow/pull/13662)

**Note**: *This issue was originally created as [ARROW-14635](https://issues.apache.org/jira/browse/ARROW-14635). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start with the superseding issue #32418 on O_DIRECT write support and review pull requests #13640 and #13662 for prior approaches. Compare direct I/O, WONTNEED, and write-cache accounting against the requirement to limit total process-plus-cache RAM during dataset writes; the work is done when an agreed mechanism prevents excessive cache growth without making the system unusable.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.