apache / apache/texera

Workflow Performance Bottleneck and Output Discrepancy in R UDF Operator + Scatter Plot Alpha Adjustment Request

Open
#3,494 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
314
Forks
187
Avg merge
1d 21h
Merged PRs (30d)
214

Description

Hi team,

I'm Ha from Steve Parker lab at U of Michigan. I'm testing local node Texera with this workflow and I found a few issues.

1. The workflow takes quite a while to get it done. This takes around 20 minutes. When using python, it only takes 5 minutes. I think sorting and R UDF operator are the bottlenecks, esp. with sorting tuples. Attached files are the workflow and raw data files with ~550k lines.

[hpap-019-metrics-csv.zip](https://github.com/user-attachments/files/20847856/hpap-019-metrics-csv.zip)

[barcode_rank_plot.json](https://github.com/user-attachments/files/20847853/barcode_rank_plot.json)

2. The number of output tuples shown on the R UDF operator should be the same as the number of input tuples, but the numbers are different.

![Image](https://github.com/user-attachments/assets/e85879a6-08c6-4231-8a31-81fd181e91c1)

3. Can we add an option to change the `alpha` value of the scatter plot?

This is the result of the python code I used with the `alpha` value being 0.2.
![Image](https://github.com/user-attachments/assets/a4be541f-10c0-43e1-b942-218da08a6253)

This is the result of the texera workflow. It is using a default `alpha` value.
![Image](https://github.com/user-attachments/assets/d0251523-854e-47ae-9657-06eb1758077d)

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the reported workflow with the attached raw data and barcode_rank_plot.json, then investigate the R UDF output count and scatter plot alpha behavior. The issue also reports a sorting and R UDF performance gap, so completion would require defining and addressing the relevant performance, output-count, and visualization discrepancies.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, r
Domain
data-visualization, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.