apache / apache/datafusion

Add Memory Profiling Functionality

Open
#14,510 10 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

When executing long-running batch processes or streaming queries, it is difficult to diagnose and monitor memory usage. People need an easy way to inspect memory consumption to understand resource usage and troubleshoot potential OOM. issues.

### Describe the solution you'd like

Implement a built-in memory profiling feature
(similar to DuckDB’s duckdb_memory() and duckdb_temporary_files() functions)
```
┌──────────────────┬────────────────────┬─────────────────────────┐
│ tag │ memory_usage_bytes │ temporary_storage_bytes │
│ varchar │ int64 │ int64 │
├──────────────────┼────────────────────┼─────────────────────────┤
│ BASE_TABLE │ 168558592 │ 0 │
│ HASH_TABLE │ 0 │ 0 │
│ PARQUET_READER │ 0 │ 0 │
│ CSV_READER │ 0 │ 0 │
│ ORDER_BY │ 0 │ 0 │
│ ART_INDEX │ 0 │ 0 │
│ COLUMN_DATA │ 0 │ 0 │
│ METADATA │ 0 │ 0 │
│ OVERFLOW_STRINGS │ 0 │ 0 │
│ IN_MEMORY_TABLE │ 0 │ 0 │
│ ALLOCATOR │ 0 │ 0 │
│ EXTENSION │ 0 │ 0 │
├──────────────────┴────────────────────┴─────────────────────────┤
│ 12 rows 3 columns │
└─────────────────────────────────────────────────────────────────┘

FROM duckdb_temporary_files();
┌────────────────────────────────┬───────────┐
│ path │ size │
│ varchar │ int64 │
├────────────────────────────────┼───────────┤
│ .tmp/duckdb_temp_storage-0.tmp │ 967049216 │
└────────────────────────────────┴───────────┘
```
that allows to query:

* The memory usage of different streams or even other objects (e.g., hash tables, intermediate results, buffers).
* Current temporary file usage, if applicable.

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

The issue names no files, tests, or entry points. Start by locating DataFusion’s existing query functions and memory or temporary-storage accounting, using DuckDB’s duckdb_memory() and duckdb_temporary_files() as behavioral references. Done means queries can expose current memory usage by category and temporary file usage, with coverage for the requested outputs.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
backend, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.