Refactor Expr architecture
- Dominant language
- C++
- Stars
- 1.9k
- Forks
- 164
- Avg merge
- 7h 31m
- Merged PRs (30d)
- 1
Description
Our current approach for creating new `Expr`s is overly complicated, as outlined here: https://github.com/h2oai/datatable/blob/master/src/core/expr/!readme.md. This complexity stems mostly from the fact that the `Expr` class which performs arithmetic on `f-expressions` is defined in pure python, and then needs to be bridged into the C++ core.
A more sane approach would be to define everything in C++, eliminating most of the "middle-man" code. In particular, the following architecture is proposed:
- C++ class `py::FExpr` to replace current python `Expr` class;
- C++ class `py::ColumnNamespace` to replace current python `FrameProxy` class;
- C++ class `dt::expr::FExpr` is a merged version of current `dt::expr::Expr` and `dt::expr::Head`. The class is virtual, with the hierarchy following that of the `Head` class;
- Each `py::FExpr` contains a `shared_ptr`;
- The `dt::expr::FExpr` class defines virtual methods for evaluation and reproing;
- The `Op` enum is removed.
## subtasks
- [x] Add support for numeric and comparison methods in `py::XObject`;
- [x] Create class `py::FExpr` (which will eventually replace the pure-python `datatable.expr.Expr`);
- [x] Create class `dt::expr::FExpr` which is a backend for `py::FExpr`;
- [x] Arrange so that new `FExpr`s can be used alongside old pure-python `Expr`s;
- [x] Create class `py::Namespace` to replace pure-python `datatable.expr.FrameProxy`;
- [ ] Convert existing `OldExpr`-based functionality into FExprs:
- [x] Frame-expr;
- [x] List-expr;
- [x] Dict-expr;
- [x] Literal exprs:
- [x] None;
- [x] bool;
- [x] int;
- [x] float;
- [x] str;
- [x] type;
- [x] range;
- [x] slice (all);
- [x] slice (numeric);
- [x] slice (string);
- [x] Column selectors `f.A` / `f[0]`;
- [x] `f.extend()`;
- [x] `f.remove()`;
- [x] Cast functions;
- [ ] `shift()`;
- [x] `ifelse()`;
- [x] `cut()`;
- [x] `qcut()`;
- [x] Arithmetic binary operators
- [x] `+`;
- [x] `-`;
- [x] `*`;
- [x] `/`;
- [x] `//`;
- [x] `%`;
- [x] `**`;
- [ ] Bitwise binary operators
- [ ] `&`;
- [ ] `|`;
- [ ] `^`;
- [ ] `<<`;
- [ ] `>>`;
- [ ] Unary operations
- [ ] `+`;
- [ ] `-`;
- [ ] `~`;
- [x] Comparison operators
- [x] `<`;
- [x] `>`;
- [x] `<=`;
- [x] `>=`;
- [x] `==`;
- [x] `!=`;
- [x] String methods
- [x] `len()`
- [x] `re_match()`;
- [ ] Reducers
- [x] `mean`,
- [x] `min`,
- [x] `max`,
- [ ] `stdev`,
- [ ] `first`,
- [ ] `last`,
- [x] `sum`,
- [x] `count`,
- [x] `count0`,
- [ ] `median`,
- [ ] `cov`,
- [ ] `corr`;
- [ ] Math functions:
- [ ] Trigonometric
- [ ] `sin`,
- [ ] `cos`,
- [ ] `tan`,
- [ ] `arcsin`,
- [ ] `arccos`,
- [ ] `arctan`,
- [ ] `arctan2`,
- [ ] `hypot`,
- [ ] `deg2rad`,
- [ ] `rad2deg`;
- [ ] Hyperbolic
- [ ] `sinh`,
- [ ] `cosh`,
- [ ] `tanh`,
- [ ] `arsinh`,
- [ ] `arcosh`,
- [ ] `arcosh`;
- [ ] Exponential
- [ ] `cbrt`,
- [ ] `exp`,
- [ ] `exp2`,
- [ ] `expm1`,
- [ ] `log`,
- [ ] `log10`,
- [ ] `log1p`,
- [ ] `log2`,
- [ ] `logaddexp`,
- [ ] `logaddexp2`,
- [ ] `pow`,
- [ ] `sqrt`,
- [ ] `square`;
- [ ] Special
- [ ] `erf`,
- [ ] `erfc`,
- [ ] `gamma`,
- [ ] `lgamma`;
- [ ] Floating
- [ ] `abs`,
- [ ] `ceil`,
- [ ] `copysign`,
- [ ] `fabs`,
- [ ] `floor`,
- [ ] `frexp`,
- [ ] `isclose`,
- [ ] `isfinite`,
- [ ] `isinf`,
- [ ] `isna`,
- [ ] `ldexp`,
- [ ] `modf`,
- [ ] `rint`,
- [ ] `sign`,
- [ ] `signbit`,
- [ ] `trunc`;
- [ ] Miscellaneous
- [ ] `clip`,
- [ ] `divmod`,
- [ ] `fmod`,
- [ ] `maximum`,
- [ ] `minimum`;
- [x] Row-functions:
- [x] `rowall`,
- [x] `rowany`,
- [x] `rowcount`,
- [x] `rowfirst`,
- [x] `rowlast`,
- [x] `rowmin`,
- [x] `rowmax`,
- [x] `rowmean`,
- [x] `rowsum`,
- [x] `rowsd`;
- [x] Documentation:
- [x] Update documentation on how to work with new FExpr infrastructure ("expr/!readme.md");
- [x] Add API documentation for the `py::Namespace` class;
- [x] Add API documentation for the `py::FExpr` class;
- [ ] Final cleanup:
- [x] Remove python class `datatable.expr.FrameProxy`;
- [ ] Remove python class `datatable.expr.Expr`;
- [ ] Remove python enum `datatable.expr.OpCodes`;
- [ ] Remove the `dt::expr::Op` enum;
- [ ] Remove the `dt::expr::OldExpr` class;
- [ ] Remove `args_registry`.
Contributor guide
Assessment
This issue has not been assessed yet.