apache / apache/arrow-rs-object-store

Slow local file reads when using GetResult.into_stream

Open
#693 7 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
322
Forks
212
Avg merge
5d 2h
Merged PRs (30d)
10

Description

**Describe the bug**
See https://github.com/apache/datafusion/issues/21450
Root cause: there's a `spawn_blocking` call per each 8KiBs read from the file, adding significant context switch overhead

**To Reproduce**
See https://github.com/apache/datafusion/issues/21446
For the tests I've used a c7a.16xlarge ec2 instance, with a trimmed down version of hits.json to 51G (original has 217 GiB), with a warm cache (by running cat hits_50.json > /dev/null)

**Expected behavior**

A more efficient implementation (e.g. tokio uses a buffer size of 2MiBs when reading files)

**Additional context**
https://github.com/apache/datafusion/pull/21478#discussion_r3078305447

Contributor guide

Open the contributing guide

Research direction

Start at GetResult.into_stream and trace the per-8KiB spawn_blocking path described in the report. Compare its read behavior with the 2MiB buffering noted for tokio, then use the linked DataFusion reproductions (issues 21446 and 21450) with the warm-cache 51G file to verify that reads no longer incur excessive context-switch overhead.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.