Improve notebooks UX for dataframes with large number of columns
- Dominant language
- Kotlin
- Stars
- 1.1k
- Forks
- 83
- Avg merge
- 4d 12h
- Merged PRs (30d)
- 30
Description
Example:
```
%use dataframe
#%%
val df = (0..100_000).toDataFrame {
"i" from { it }
"test" from { it }
}
df
#%%
val res = df.pivot { i }.with { test }
```
Few things are unclear
1. You actually can use res variable in the next cell, only without typesafe wrapper - message should be improved
2. Need a way to suppress this error, like that codegen callback shouldn't run for this variable
3. Actual error message looks fixable, although i doubt even with this fixed we won't get OOM or other performance problem
```
Caused by: org.jetbrains.kotlinx.jupyter.exceptions.ReplCompilerException: class org.jetbrains.kotlin.com.intellij.psi.impl.file.PsiLargeTextFileImpl cannot be cast to class org.jetbrains.kotlin.psi.KtFile (org.jetbrains.kotlin.com.intellij.psi.impl.file.PsiLargeTextFileImpl and org.jetbrains.kotlin.psi.KtFile are in unnamed module of loader 'app')
at org.jetbrains.kotlinx.jupyter.repl.impl.JupyterCompilerImpl.compileSync(JupyterCompilerImpl.kt:151)
at org.jetbrains.kotlinx.jupyter.repl.impl.InternalEvaluatorImpl.eval(InternalEvaluatorImpl.kt:126)
at org.jetbrains.kotlinx.jupyter.repl.impl.CellExecutorImpl.execute_W38Nk0s$lambda$0$1(CellExecutorImpl.kt:95)
at org.jetbrains.kotlinx.jupyter.repl.impl.ReplForJupyterImpl.withHost(ReplForJupyterImpl.kt:730)
at org.jetbrains.kotlinx.jupyter.repl.impl.CellExecutorImpl.execute-W38Nk0s(CellExecutorImpl.kt:93)
at org.jetbrains.kotlinx.jupyter.repl.execution.CellExecutor.execute-W38Nk0s$default(CellExecutor.kt:14)
at org.jetbrains.kotlinx.jupyter.repl.impl.CellExecutorImpl$ExecutionContext.execute(CellExecutorImpl.kt:307)
at org.jetbrains.kotlinx.dataframe.jupyter.ExecuteKt.execute(execute.kt:15)
```
Another problem is table rendering. Worth to check how other handle it. Some ideas: cut number of displayed columns, automatically switch to HTML/text/transposed view (at least makes sense for DataRow)
https://github.com/user-attachments/assets/0b353f18-2e69-44d5-92ec-480ed9c23546
Contributor guide
Research direction
Reproduce the notebook example with the large dataframe and inspect the reported entry points, especially JupyterCompilerImpl.kt:151, InternalEvaluatorImpl.kt:126, CellExecutorImpl.kt, and execute.kt. Determine the intended behavior for the compiler message, code-generation suppression, and table rendering; the issue is done only when those outcomes are defined and verified without the current failure or unusable display.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- jupyter, kotlin
- Domain
- data, developer-experience
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100