havakv / havakv/pycox

PyCOX data leaking

Open
#153 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
995
Forks
203
PR merge metrics
No merged PRs in 30d

Description

Hi,

I was working on cross validation/ splitting data using different seed points and then train a PyCOX model before averaging the result. I tried to fix the data leakage y removing the model/log/ and other variables where there can be some data leakage. Also initializing model every time in the loop and emptying the cache also. But still I have some data leakage.

Can anyone help me how to fix/reduce this?

Please check the code below:

for seed in SEEDS:
data, target = X1,Z1
X_train1,X_test,Z_train1,Z_test = train_test_split(data, target, test_size=0.25, random_state=seed)
X_train,X_val,Z_train,Z_val = train_test_split(X_train1,Z_train1, test_size=0.25, random_state=seed)
val = X_val, Z_val
get_target = lambda df: (df['AVAL_PFS'].to_numpy(dtype ='float32'), df['EVENT_PFS'].to_numpy(dtype ='int32'))
durations_test, events_test = get_target(Z_test)

model2 = CoxPH(net2, device=torch.device('cuda:0'))
model2.optimizer.set_lr(0.002)

log = model2.fit(X_train, Z_train, batch_size, epochs, callbacks,verbose, val_data=val, val_batch_size=batch_size)

_ = model2.compute_baseline_hazards()

surv = model2.predict_surv_df(X_test)
ev = EvalSurv(surv, durations_test, events_test, censor_surv='km')
score=ev.concordance_td()
cv_score.append(score)
del log,model2,optimizer,X_train,X_test,Z_train, val,surv, durations_test, events_test,score
torch.cuda.empty_cache()'

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue names no files or tests; start with the cross-validation loop shown and trace train_test_split, CoxPH.fit, compute_baseline_hazards, and EvalSurv across iterations. Determine which model or data state is shared between splits, then demonstrate independent evaluation with a reproducible comparison showing that the leakage is removed.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.