apache / apache/datafusion

Further improve datafusion-cli memory usage if we setting huge number for maxrow size.

Open
#14,810 6 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

This is a follow-up for the bellow comments:

https://github.com/apache/datafusion/pull/14766#discussion_r1965294394

### Describe the solution you'd like

Streaming datafusion-cli the print batch progress.

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the discussion linked from the issue, especially the comments on pull request #14766, and trace how datafusion-cli handles maxrow and batch printing. The requested outcome is streaming batch progress so a huge maxrow setting does not cause excessive memory use; the issue does not name specific files or tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
cli
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.