Frame capture improvements
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
Looking at a simple dask tutorial
```python
ddf = dd.read_parquet(
"s3://dask-data/nyc-taxi/nyc-2015.parquet/part.*.parquet",
columns=["passenger_count", "tip_amount"],
storage_options={"anon": True},
)
result = ddf.groupby("passenger_count").tip_amount.mean().compute()
```
On the scheduler this generates a bunch of tasks on the scheduler with names like `series-grouby-chunk-agg`, `truediv` and `finalize`. I can kinda piece things together and see that those make sense, but for more complex examples the task names often become meaningless to me. When something breaks or is a bottleneck, its hard to quickly reason about what in my code generated these tasks, and what I might do to fix it.
Could we do something better with the way we capture code and link it to task groups (or groups of task groups?)
I'm dreaming of a solution where I can click on one of the `read_parquet` tasks and see something that shows that it was generated because of this call in my code .
```python
ddf = dd.>>read_parquet<<(
"s3://dask-data/nyc-taxi/nyc-2015.parquet/part.*.parquet",
columns=["passenger_count", "tip_amount"],
storage_options={"anon": True},
)
```
Not the best example as the name matches the function call but hopefully you get the idea.
It's possible this becomes totally impractical for more complex operations or we just can't group the tasks up in a way that makes sense. I don't have a good enough grasp on how the tasks are generated to know.
Contributor guide
Assessment
This issue has not been assessed yet.