Kotlin / Kotlin/dataframe

Improve notebooks UX for dataframes with large number of columns

Open
#1,579 0 comments 2 reactions 0 assignees View on GitHub
bug
Dominant language
Kotlin
Stars
1.1k
Forks
83
Avg merge
4d 12h
Merged PRs (30d)
30

Description

Example:
```
%use dataframe
#%%
val df = (0..100_000).toDataFrame {
"i" from { it }
"test" from { it }
}
df
#%%
val res = df.pivot { i }.with { test }
```

Few things are unclear
Image

1. You actually can use res variable in the next cell, only without typesafe wrapper - message should be improved
2. Need a way to suppress this error, like that codegen callback shouldn't run for this variable
3. Actual error message looks fixable, although i doubt even with this fixed we won't get OOM or other performance problem
```
Caused by: org.jetbrains.kotlinx.jupyter.exceptions.ReplCompilerException: class org.jetbrains.kotlin.com.intellij.psi.impl.file.PsiLargeTextFileImpl cannot be cast to class org.jetbrains.kotlin.psi.KtFile (org.jetbrains.kotlin.com.intellij.psi.impl.file.PsiLargeTextFileImpl and org.jetbrains.kotlin.psi.KtFile are in unnamed module of loader 'app')
at org.jetbrains.kotlinx.jupyter.repl.impl.JupyterCompilerImpl.compileSync(JupyterCompilerImpl.kt:151)
at org.jetbrains.kotlinx.jupyter.repl.impl.InternalEvaluatorImpl.eval(InternalEvaluatorImpl.kt:126)
at org.jetbrains.kotlinx.jupyter.repl.impl.CellExecutorImpl.execute_W38Nk0s$lambda$0$1(CellExecutorImpl.kt:95)
at org.jetbrains.kotlinx.jupyter.repl.impl.ReplForJupyterImpl.withHost(ReplForJupyterImpl.kt:730)
at org.jetbrains.kotlinx.jupyter.repl.impl.CellExecutorImpl.execute-W38Nk0s(CellExecutorImpl.kt:93)
at org.jetbrains.kotlinx.jupyter.repl.execution.CellExecutor.execute-W38Nk0s$default(CellExecutor.kt:14)
at org.jetbrains.kotlinx.jupyter.repl.impl.CellExecutorImpl$ExecutionContext.execute(CellExecutorImpl.kt:307)
at org.jetbrains.kotlinx.dataframe.jupyter.ExecuteKt.execute(execute.kt:15)
```

Another problem is table rendering. Worth to check how other handle it. Some ideas: cut number of displayed columns, automatically switch to HTML/text/transposed view (at least makes sense for DataRow)

https://github.com/user-attachments/assets/0b353f18-2e69-44d5-92ec-480ed9c23546

Contributor guide

Open the contributing guide

Research direction

Reproduce the notebook example with the large dataframe and inspect the reported entry points, especially JupyterCompilerImpl.kt:151, InternalEvaluatorImpl.kt:126, CellExecutorImpl.kt, and execute.kt. Determine the intended behavior for the compiler message, code-generation suppression, and table rendering; the issue is done only when those outcomes are defined and verified without the current failure or unusable display.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter, kotlin
Domain
data, developer-experience
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.