queryverse / queryverse/Query.jl

Proposal: easier column extraction and data cleaning

Open
#146 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Julia
Stars
403
Forks
48
Avg merge
3d 6h
Merged PRs (30d)
6

Description

I was thinking that something like the @df macro in StatPlots would benefit many different packages, by allowing normal arrays from the output of a query to be fed directly to a function (especially as it can be done without even explicitly collecting the query, see here). What I was wondering is whether something similar could live in Query as well. I'm thinking about a macro of the style:

@replace_complete_cols df f(_..a, _..b, _..c .+ 1)

which would replace the _..s expression with the respective columns converted to a regular Array (it would exclude rows where a column that is being used is missing data). There are two more tools that would be helpful to implement this functionality and would go well together with it:

  • a @dropna stand-alone macro that would filter rows with no missing values
  • as mentioned here, the possibility to have a tuple in a @select statement, which could then be collected as a tuple of Arrays. Without that, selecting an arbitrary number of columns is a bit cumbersome (I haven't found a way of selecting multiple columns with a NamedTuple iterator because there doesn't seem to be a type stable way of generating a NamedTuple without manually typing each element, whereas list comprehension works just fine for tuples).

Do you believe that this kind of macro belongs to Query.jl or should it live somewhere else?
Also, what syntax would you think is best? What I put here is pretty much a placeholder.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing Query.jl's existing @select behavior and how StatPlots' @df macro handles column extraction. Compare the proposed @replace_complete_cols and @dropna macros with tuple selection in @select, then determine whether the API and syntax belong in Query.jl. Done would require an agreed design and defined behavior for missing rows, but the issue does not specify an implementation location or tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.