matloff / matloff/qeML

grouped data

Open
#11 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
R
Stars
46
Forks
15
PR merge metrics
No merged PRs in 30d

Description

With grouped data it is important that if one row of a group is in the training set other then other rows in that group cannot be in the test set That is instead of sampling individual rows sample groups. This link shows an example and there is another example further down here.

https://stackoverflow.com/questions/71087864/how-to-keep-grouped-variables-together-in-training-and-test-data

Perhaps allow the `holdout=` argument to be a vector of indexes or provide for a `group=` argument. The first possibility would allow other schemes as well whereas the second is easier for the user in this situation but does not allow for unanticipated sampling schemes. It would be possible to have both, of course.

I am currently kludging it using this where the example is iris assuming each successive 10 rows forms a group.

```
# iris where each successive 10 rows forms a group
library(qeML)
set.seed(123)

# create grouping variable
grp <- rep(1:15, each = 10)

# set holdout indexes so that if a row is in test or is in train then others in group are too
holdout <- which(grp %in% sample(15, 3))

# kludge it by redefining sample within qeKNN to return the indexes we want
trace(qeKNN, quote(sample <- function(x, holdout) holdout))
qeKNN(iris, "Species", holdout = holdout)
untrace(qeKNN)
```

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with qeKNN and its holdout handling, using the issue's grouped iris example and linked Stack Overflow discussion as context. Compare supporting grouped sampling with accepting explicit holdout indexes; done means rows from the same group cannot be split between training and test sets.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.