JuliaAI / JuliaAI/ScientificTypes.jl

Towards a more efficient `schema` methods for row-based tables

Open
#127 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Julia
Stars
98
Forks
7
PR merge metrics
No merged PRs in 30d

Description

The issue has been raised that schema, applied to a table, currently has to concretely manifest each column, as way of extracting it's scitype, which for row-based tables is inefficient. The reason for the current implementation is that, in general, the column element scitype cannot be inferred from column element machine type.

Here are some details so that someone interested can explore a workaround, which I think is certainly possible.

At present (and this might change) the only time the scitype of an array A cannot be determined from the machine type is if

eltype(A) <: CategoricalArrays.CategoricalValue    

This is because the scitype depends on: (i) whether or the pool is ordered, and (ii) the number of levels. Neither of these are in the machine type - they must be extracted from an instance. However, it is safe to assume that all elements have the same scitype, because it is very unusual for an array to have inhomogeneous pools (the CategoricalPool contains the order/levels information). Indeed, CategoricalArrays goes to great lengths to ensure creating of such arrays is difficult. Under this assumption, one can therefore compute the scitype of A by looking at just the first element (which for Tables, means looking just at the first row).

cc @OkonSamuel

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Locate the table-oriented schema implementation and trace how it currently manifests columns for row-based tables. Study the CategoricalArrays.CategoricalValue case and evaluate computing its scitype from the first row while preserving ordered and level information; done means schema results remain correct with less column materialization.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.