Incorrect output for nested reducers

Nesting aggregation calls sometimes produces incorrect results

```py
>>> from datatable import dt, f
>>> DT = dt.Frame([1, 2])
>>> DT[:, dt.mean(dt.sum(f.C0))]
| C0
| float64
-- + ----------
0 | 2.8823e+18
[1 row x 1 column]
>>> DT[:, dt.sum(dt.mean(f.C0))]
| C0
| float64
-- + -----------
0 | 2.31584e+77
[1 row x 1 column]
```

The expected behavior is
```py
>>> DT[:, dt.mean(dt.sum(f.C0))]
| C0
| float64
-- + -----
0 | 3
[1 row x 1 column]
>>> DT[:, dt.sum(dt.mean(f.C0))]
| C0
| float64
-- + -------
0 | 1.5
[1 row x 1 column]
```

datatable version: 1.1
python version: 3.9
operating system: linux