mars-project / mars-project/mars

[Discussion] Enhance evaluation module to facilitate JIT

Open
#2,827 3 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

mod: optimization needs discussions proposal
Dominant language
Python
Stars
2.7k
Forks
325
PR merge metrics
No merged PRs in 30d

Description

Mars implements `DataFrame.eval` and collects operands (at https://github.com/mars-project/mars/blob/master/mars/optimization/logical/tileable/arithmetic_query.py) that can fit into a string expression applicable for pandas eval. While improving efficiency, this implementation has drawbacks.

1. When meeting non-string column index type, for instance, `MultiIndex`, `eval` is not supported.
2. Non-arithmetic chunk-by-chunk operands not well supported.
3. Tensor fusion not supported.

To handle these issues, current implementation of optimization need to be enhanced. Instead of passing expression strings, an expression DAG with fused expressions need to be added.

We may use Mars Expression DAG itself to represent those evaluation DAGs. An evaluation expression starts with a `Fetch` node accepting the chunk itself and outputs one or more chunks as results. All acceptable operations inside the evaluation DAG must be chunk-by-chunk operands.

After generating evaluation DAGs, related operands are then condensed into a `Evaluate` operand with a `evaluation_dag` operand. When submitted to a supervisor, cluster-based optimization can also be made before tiling. After tiling this operand, a chunk-to-chunk plan is generated and then executed.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with mars/optimization/logical/tileable/arithmetic_query.py and the DataFrame.eval implementation. Trace how operands are collected and how the optimization path reaches submission and tiling. Done means evaluation is represented as an expression DAG with fused chunk-by-chunk operations, supports non-string indexes and tensor fusion, and produces the described Evaluate chunk-to-chunk plan.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data-engineering, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.