mars-project / mars-project/mars
[Discussion] Enhance evaluation module to facilitate JIT
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.7k
- Forks
- 325
- PR merge metrics
- No merged PRs in 30d
Description
Mars implements `DataFrame.eval` and collects operands (at https://github.com/mars-project/mars/blob/master/mars/optimization/logical/tileable/arithmetic_query.py) that can fit into a string expression applicable for pandas eval. While improving efficiency, this implementation has drawbacks.
1. When meeting non-string column index type, for instance, `MultiIndex`, `eval` is not supported.
2. Non-arithmetic chunk-by-chunk operands not well supported.
3. Tensor fusion not supported.
To handle these issues, current implementation of optimization need to be enhanced. Instead of passing expression strings, an expression DAG with fused expressions need to be added.
We may use Mars Expression DAG itself to represent those evaluation DAGs. An evaluation expression starts with a `Fetch` node accepting the chunk itself and outputs one or more chunks as results. All acceptable operations inside the evaluation DAG must be chunk-by-chunk operands.
After generating evaluation DAGs, related operands are then condensed into a `Evaluate` operand with a `evaluation_dag` operand. When submitted to a supervisor, cluster-based optimization can also be made before tiling. After tiling this operand, a chunk-to-chunk plan is generated and then executed.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with mars/optimization/logical/tileable/arithmetic_query.py and the DataFrame.eval implementation. Trace how operands are collected and how the optimization path reaches submission and tiling. Done means evaluation is represented as an expression DAG with fused chunk-by-chunk operations, supports non-string indexes and tensor fusion, and produces the described Evaluate chunk-to-chunk plan.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100