lincc-frameworks / lincc-frameworks/nested-pandas
NestedFrame.query() should handle mixed base and nested columns without erroring
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 26
- Forks
- 8
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 9
Description
Today, NestedFrame.query checks the expression it's given and raises ValueError if it mixes nested and base columns. This is because in order to handle such expressions correctly, it would need to tease out the sub-expressions that are strictly against the nested columns (by traversing the abstract syntax tree of the input expression), apply and re-pack into an intermediate result, and then apply the base column expressions to this intermediate result.
In an expression like a > 2 & nested.flux > 50, for example, the user would expect the resulting NestedFrame to have no a values which were <= 2 and no nested.flux values which were <= 50. And in an expression like a > 2 | nested.flux > 50, the user would still expect to retain rows where a <= 2 so long as it had some nested.flux > 50, but within those rows, they wouldn't expect to see any nested.flux <= 50. For those rows where a > 2, though, they'd expect to see all the nested.flux rows. In other words, as soon as there is mixed-level expression, the nested rows sometimes need to be queried and repacked before continuing, or at least that should be the final effect.
Logically, if there was a method to unpack all nests and broadcast all base columns across them, then we would take the result of self.eval(expr) and do something like self.flatten_all().loc[result].repack_all(), but this would likely not be performant.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at NestedFrame.query, where mixed nested and base-column expressions currently raise ValueError, and trace how the expression is evaluated and nested results are repacked. Use the AND and OR examples in the issue as behavioral checks; done means mixed expressions work without error and retain exactly the expected base and nested values.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100