“EXPLAIN / profiling” support for [i, j, by] to diagnose performance bottlenecks
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 25/100
- Issue type
- Feature
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- r
- Domain
- data, performance
Research direction
The issue names no files, tests, or entry points. Start by reviewing how DT[i, j, by] handles filtering, grouping, joins, keys or indices, materialization, and copies. Define the diagnostics and acceptance criteria for the proposed explain or profiling mode.
Written by the indexing model from the issue text.
Description
Problem
A common difficulty for users is understanding why a particular data.table query is slow.
When writing expressions like:
DT[ i , j , by]
there is currently no way to determine:
Whether time is spent in i filtering, j computation, or by grouping
Whether a key / index is actually being used
Whether grouping is triggering expensive materialization
Whether unexpected memory copies are occurring
Whether a join is using a fast path or falling back to a slower path
Most users rely on:
system.time(DT[i, j, by])
which measures only the total time and gives no insight into the internal bottleneck.
This makes optimization and debugging of complex workflows difficult, especially for users familiar with SQL-style tools like EXPLAIN.
Proposed Idea
Introduce an optional profiling / explain mode for data.table operations, conceptually similar to SQL’s EXPLAIN.
For example:
explain( DT[x > 5, .(m = mean(y)), by = z] )
or
options(datatable.explain = TRUE) DT[x > 5, .(m = mean(y)), by = z]
This could output structured diagnostics such as:
data.table EXPLAIN
Rows scanned: 5,000,000
Rows matched in i: 1,240,532
Groups formed by by: 2,134
Time spent:
i (filter): 120 ms
by (grouping): 340 ms
j (compute): 90 ms
Keys / indices used: YES (key: z)
Materialization: NO
Memory copies: 1 shallow copy
- Dominant language
- R
- Stars
- 3.9k
- Forks
- 1.1k
- Avg merge
- 14h 4m
- Merged PRs (30d)
- 4
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from Rdatatable/data.table
-
as.data.table() recurses without end on a survival::Surv object (or any data.frame carrying one) Open
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
Rdatatable/data.table#7887 ·
-
consistency tests
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Rdatatable/data.table#7853 · 3 comments ·
-
internals
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Rdatatable/data.table#6938 · 1 comment ·
-
encoding fread
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Rdatatable/data.table#5179 · 8 comments ·
-
documentation programming
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Rdatatable/data.table#3199 · 3 comments ·
All issues in Rdatatable/data.table
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
r-lib/pkgdepends#485 · 3 comments ·
-
Difficulty 1/5 Under an hour Newbie friendliness 92/100
-
beginners blocker
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enviPathR OpenBuild Error Build OK Build Warning policies-accepted pre-review precheck-passed
Difficulty 1/5 Under an hour Newbie friendliness 84/100
Bioconductor/BiocContributions#207 · 6 comments ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
datacarpentry/semester-biology#1255 ·