lincc-frameworks / lincc-frameworks/nested-pandas

Make it easier to run aggregations over nested elements in nf.eval, nf.query and nf.nested.nest

Open
#155 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement interface performance
Dominant language
Python
Stars
26
Forks
8
Avg merge
2d 2h
Merged PRs (30d)
9

Description

**Feature request**

Today, we have these ways to aggregate a single nested column values:
- `nf.reduce(np.mean, "lc.mag")` - good, but not cheap and requires to join the output back to the frame
- `nf.eval("lc.mag.groupby(by=lc.mag.index).mean()")` - expansive and not intuitive

It would be nice if we can develop an easier way of doing such aggregations. Options I see:

1. Currently, we can do `nf.eval("lc.mag.mean()")` / `nf["lc.mag"].mean()`, but it would output the aggregation over all the flat values, which is, especially in the first case, not intuitive. We can redefine it.
2. Add special interface for nested aggregations with `.nest` accessor, e.g. `nf.lc.nest.mean()` would return `nf.shape[0]` mean values.
3. Add special methods which would work in `eval/query` environment only, e.g. `nf.eval("lc.mag.nest_mean()")`

However I'm not sure how we'd make all these performant, it looks like `pyarrow` provides almost zero tooling for that. Maybe we can use things like [`numpy.ufunc.reduceat`](https://numpy.org/doc/stable/reference/generated/numpy.ufunc.reduceat.html#numpy.ufunc.reduceat) and [`scipy.ndimage.mean`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.ndimage.mean.html#scipy.ndimage.mean).

**Before submitting**
Please check the following:

- [x] I have described the purpose of the suggested change, specifying what I need the enhancement to accomplish, i.e. what problem it solves.
- [x] I have included any relevant links, screenshots, environment information, and data relevant to implementing the requested feature, as well as pseudocode for how I want to access the new functionality.
- [ ] If I have ideas for how the new feature could be implemented, I have provided explanations and/or pseudocode and/or task lists for the steps.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The request names nf.eval, nf.query, and nf.nested.nest as entry points; start by tracing their existing aggregation behavior and reviewing the nested-column representation. The aggregation interface, semantics, and performance approach are undecided, so completion requires agreeing on a concrete API and defining tests for the expected per-row results.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, pandas, python
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.