Trouble training with 2 GPUs

• PyTorch-Forecasting version: 0.8.3
• PyTorch version: 1.7.1
• Python version: 3.7.6
• Operating System: Ubuntu 18.04

Similarities

I notice this is similar to #103 and #215, which were seemingly resolved. (?)

Also I should mention I'm not sure if this is perhaps a PL issue.

Expected behavior

I would like to train a model across 2 GPUs in order to speed up training. I just set the gpus flag of Pytorch Lightning's Trainer constructor to 2: gpus=2.

Accelerator

With the flag accelerator='ddp', I get one error, while with the flag accelerator='dp', the kernel is perpetually busy but does not begin training. With accelerator='ddp_spawn', I get a different error. I think the one to use for me would be ddp spawn, since DDP is not possible in Jupyter Notebook.

I think the accelerator I need for my case is ddp spawn.

Actual behavior

With ddp_spawn, the following error occurs: TypeError: can't pickle torch._C.Generator objects

With ddp, the kernel is perpetually busy and training doesn't start (presumably because I'm using a notebook).

With dp, the following error occurs:
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1!

I suspect this one is because of the way my GPUs are I shouldn't be using dp but I'm really not sure. I figured I'd try all my options.

Is ddp the only one that works at this time? Is there any way of using 2 GPUs in a notebook?

Thanks for your help.

My understanding is that the sampler doesn't actually matter because PL overwrites it with DistributedSampler when you instantiate Trainer with gpus > 1, but I might be wrong so I included it below anyhow.

Code to reproduce the problem

### fit network
trainer.fit(
    tft,
    train_dataloader=train_dataloader,
    val_dataloaders=val_dataloader,
)

This call is where all three issues occur. Please see the training and trainer initialization below;

### Split up dataset appropriately

max_prediction_length = VALIDATION_LENGTH + TEST_LENGTH # Number of hours NOT shown to network

training_cutoff = data["time_idx"].max() - max_prediction_length # which hour value to stop training
parameters = data.columns[:3].tolist()


## Create training dataset
training = TimeSeriesDataSet(
    data[lambda x: x.time_idx <= training_cutoff],
    time_idx="time_idx",
    target="occupancy",
#     group_ids=parameters,
    group_ids=[parameters[0]],
    min_encoder_length=MIN_ENCODER_LENGTH,
#     min_encoder_length=VALIDATION_LENGTH,
    max_encoder_length=MAX_ENCODER_LENGTH,
    min_prediction_length=MIN_PREDICTION_LENGTH,
    max_prediction_length=max_prediction_length,
#     static_categoricals=parameters,
    static_categoricals=[parameters[0]],
#     static_reals=[],
    static_reals=parameters[1:3],
    time_varying_known_categoricals=["day_wk", "time_day"],
    variable_groups={},
    time_varying_known_reals=["time_idx"],
    time_varying_unknown_categoricals=[],
    time_varying_unknown_reals=["occupancy"],
    target_normalizer=GroupNormalizer(transformation=None, center=False),
    randomize_length=True, # should configure specific beta distribution at some pt
    add_relative_time_idx=True,
    add_target_scales=True,
    add_encoder_length=True, # randomize time-length of samples between MIN and MAX LENGTH defined
)


# create validation set (predict=True) which means to predict the last max_prediction_length points in time
# for each series
val_cutoff = data["time_idx"].max() - TEST_LENGTH

validation = TimeSeriesDataSet.from_dataset(
    training,
    data[lambda x: x.time_idx <= val_cutoff],
    predict=True,
    min_prediction_idx=training_cutoff + 1, # since max pred len of train > val len, set min val index
    max_prediction_length=VALIDATION_LENGTH,
    min_prediction_length=1,
    stop_randomization=True,
)


## Configure sampling
batch_sampler = torch.utils.data.RandomSampler(
    training,
    replacement=True,
    num_samples=2500, # randomly sample 20000 mini-timeseries' from various channels, length determined by encoder length
    generator=torch.Generator(),
)

batch_sampler = torch.utils.data.BatchSampler(batch_sampler, batch_size=BATCH_SIZE//2, drop_last=False)



# create dataloaders for model
train_dataloader = training.to_dataloader(
    train=True,
    batch_size=BATCH_SIZE,
    num_workers=NUM_WORKERS,
    # don't use any sampler because pl automatically samples for multi-GPU training
    batch_sampler=batch_sampler,
#     batch_sampler=sampler,
)


# train_dataloader = torch.utils.data.DataLoader(training, batch_size=BATCH_SIZE)

val_dataloader = validation.to_dataloader(
    train=False,
    batch_size=BATCH_SIZE * 8,
    num_workers=NUM_WORKERS,
)


### configure network and trainer
early_stop_callback = EarlyStopping(monitor="val_loss", min_delta=1e-9, patience=10, verbose=True, mode="min")
lr_logger = LearningRateMonitor()  # log the learning rate
logger = TensorBoardLogger("lightning_logs")  # logging results to a tensorboard
print_updates = PrintMetrics(10) # permanently print losses every 10 batches

trainer = pl.Trainer(
    max_epochs=100,
#     max_epochs=1, # test
    gpus=2, # use all GPUs,
#     accelerator='ddp_spawn', # cant use ddp in jupyter
    accelerator='dp',
    accelerator='dp',
    weights_summary='top',
#     gradient_clip_val=0.01,
    limit_train_batches=0.1,
#     shuffle=True,
#     fast_dev_run=True,  # comment in to check that networkor dataset has no serious bugs
    callbacks=[lr_logger, early_stop_callback, print_updates],
    logger=logger,
)

tft = TemporalFusionTransformer.from_dataset(
    training,
    learning_rate=0.002,
    lstm_layers=2,
    hidden_size=128,
    attention_head_size=4,
    dropout=0.3,
    hidden_continuous_size=128,
    output_size=7,  # 7 quantiles by default
    loss=QuantileLoss(),
    log_interval=10,  # uncomment for learning rate finder and otherwise, e.g. to 10 for logging every 10 batches
    reduce_on_plateau_patience=5,
)

P.S. Loving this package. Getting some good results!

Edit: swapped error cases by accident

Start by reproducing the failure at trainer.fit with the shown Trainer accelerator settings and the custom RandomSampler in the notebook setup. Compare the ddp_spawn, ddp, and dp errors against the stated PyTorch and PyTorch-Forecasting versions. Done means identifying a supported two-GPU notebook configuration or documenting the blocking incompatibility and its cause.