anthony-wang / anthony-wang/CrabNet

CrabNet matbench results - possibly neglecting 25% of the training data it could have used

Aperta
#19 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Python
Stelle
132
Fork
42
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

@anthony-wang,

In the [CrabNet matbench notebook](https://github.com/materialsproject/matbench/blob/main/benchmarks/matbench_v0.1_CrabNet/notebook.ipynb), it does train/val/test splits. However, if https://github.com/anthony-wang/CrabNet/issues/15#issuecomment-925519683 is correct such that the validation data (i.e. `val.csv`) doesn't contribute to hyperparameter tuning, then that 25% of the training data is essentially getting thrown away, correct?

In other words, the CrabNet results are based on only 75% of the training data compared to what the other `matbench` models use for training. From what I understand, the train/val/test split in the context of `matbench` only really makes sense if you're doing hyperparameter optimization in a nested CV scheme, as follows:

(Source: https://hackingmaterials.lbl.gov/automatminer/advanced.html)

To correct this, I think all that needs to be done is change:
```python
#split_train_val splits the training data into two sets: training and validation
def split_train_val(df):
df = df.sample(frac = 1.0, random_state = 7)
val_df = df.sample(frac = 0.25, random_state = 7)
train_df = df.drop(val_df.index)

return train_df, val_df
```
to
```python
#split_train_val splits the training data into two sets: training and validation
def split_train_val(df):
train_df = df.sample(frac = 1.0, random_state = 7)
val_df = df.sample(frac = 0.25, random_state = 7)

return train_df, val_df
```
which makes it so there's data bleeding between `train_df` and `val_df`, but `val_df` ends up being essentially just a dummy dataset so that CrabNet doesn't error out when a `val.csv` isn't available.

Sterling

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.