JuliaAI / JuliaAI/MLJBase.jl

Handling of missings

Open
#97 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

design discussion
Dominant language
Julia
Stars
163
Forks
46
Avg merge
1d 18h
Merged PRs (30d)
5

Description

Not sure if it's the appropriate place to post this; it looks like we could improve the situations when dealing with missings. Here's a simple example of something that's not ideal:

  1. y has some missings
  2. X doesn't
  3. we drop the rows of y and X that have missings
  4. we feed that in a machine
  5. the machine fails because the model doesn't handle Union{Missing,Float64}

of course there are no missings in y anymore. Interestingly, the scientific type is correct so that could be exploited but the machine type has Union{Missing,...}

Maybe we could help with:

  • provide a dropmissing function which "re-coerces"
  • have a mechanism in fit which checks whether the type and scientific type match in terms of missing, if they don't and ST has it right, maybe the data can be re-coerced

thoughts?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing MLJBase's existing handling of missing values, scientific types, and the fit path described in the issue. The scope is not settled: compare the proposed dropmissing re-coercion with fit-time checking, then clarify the intended behavior and add tests demonstrating that cleaned data no longer retains an incompatible missing-value type.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.