Workflow Performance Bottleneck and Output Discrepancy in R UDF Operator + Scatter Plot Alpha Adjustment Request
- Dominant language
- Scala
- Stars
- 314
- Forks
- 187
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 214
Description
Hi team,
I'm Ha from Steve Parker lab at U of Michigan. I'm testing local node Texera with this workflow and I found a few issues.
1. The workflow takes quite a while to get it done. This takes around 20 minutes. When using python, it only takes 5 minutes. I think sorting and R UDF operator are the bottlenecks, esp. with sorting tuples. Attached files are the workflow and raw data files with ~550k lines.
[hpap-019-metrics-csv.zip](https://github.com/user-attachments/files/20847856/hpap-019-metrics-csv.zip)
[barcode_rank_plot.json](https://github.com/user-attachments/files/20847853/barcode_rank_plot.json)
2. The number of output tuples shown on the R UDF operator should be the same as the number of input tuples, but the numbers are different.

3. Can we add an option to change the `alpha` value of the scatter plot?
This is the result of the python code I used with the `alpha` value being 0.2.

This is the result of the texera workflow. It is using a default `alpha` value.

Contributor guide
Research direction
Start by reproducing the reported workflow with the attached raw data and barcode_rank_plot.json, then investigate the R UDF output count and scatter plot alpha behavior. The issue also reports a sorting and R UDF performance gap, so completion would require defining and addressing the relevant performance, output-count, and visualization discrepancies.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, r
- Domain
- data-visualization, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100