JuliaAI / JuliaAI/ScientificTypes.jl
Towards a more efficient `schema` methods for row-based tables
Nobody has claimed this yet.
- Dominant language
- Julia
- Stars
- 98
- Forks
- 7
- PR merge metrics
- No merged PRs in 30d
Description
The issue has been raised that schema, applied to a table, currently has to concretely manifest each column, as way of extracting it's scitype, which for row-based tables is inefficient. The reason for the current implementation is that, in general, the column element scitype cannot be inferred from column element machine type.
Here are some details so that someone interested can explore a workaround, which I think is certainly possible.
At present (and this might change) the only time the scitype of an array A cannot be determined from the machine type is if
eltype(A) <: CategoricalArrays.CategoricalValue
This is because the scitype depends on: (i) whether or the pool is ordered, and (ii) the number of levels. Neither of these are in the machine type - they must be extracted from an instance. However, it is safe to assume that all elements have the same scitype, because it is very unusual for an array to have inhomogeneous pools (the CategoricalPool contains the order/levels information). Indeed, CategoricalArrays goes to great lengths to ensure creating of such arrays is difficult. Under this assumption, one can therefore compute the scitype of A by looking at just the first element (which for Tables, means looking just at the first row).
cc @OkonSamuel
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Locate the table-oriented schema implementation and trace how it currently manifests columns for row-based tables. Study the CategoricalArrays.CategoricalValue case and evaluate computing its scitype from the first row while preserving ordered and level information; done means schema results remain correct with less column materialization.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- data
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100