deepspeedai / deepspeedai/DeepSpeed

How to properly use `IterableDataset`, with DeepSpeed ?

Open
#1,018 0 comments 4 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
43.1k
Forks
5k
Avg merge
4d 15h
Merged PRs (30d)
112

Description

This is a follow up on this issue

https://github.com/microsoft/DeepSpeed/issues/285

Where the same question was asked, and the users was pointed to the bing bert example [ https://github.com/microsoft/DeepSpeedExamples/tree/fd6fb5148ccf5c9ce222432006f1d93806187cd9/bing_bert ]

From looking at the bing bert example, the data generator is not loaded with deepspeed.initialize (I previously tried putting my IterableDataset instance in deepspeed.initialize, but it gave an error)

    model.network, optimizer, _, _ = deepspeed.initialize(
        args=args,
        model=model.network,
        model_parameters=optimizer_grouped_parameters)

https://github.com/microsoft/DeepSpeedExamples/blob/fd6fb5148ccf5c9ce222432006f1d93806187cd9/bing_bert/deepspeed_train.py#L438

Instead, it uses a pytorch DataLoader using a sampler from torch.utils.data.distributed.DistributedSampler

def get_dataloader(args, dataset: Dataset, eval_set=False):
    if args.local_rank == -1:
        train_sampler = RandomSampler(dataset)
    else:
        train_sampler = DistributedSampler(dataset)
    return (x for x in
            DataLoader(dataset,
                       batch_size=args.train_micro_batch_size_per_gpu //
                       2 if eval_set else args.train_micro_batch_size_per_gpu,
                       sampler=train_sampler,
                       num_workers=args.config['training']['num_workers']))

https://github.com/microsoft/DeepSpeedExamples/blob/fd6fb5148ccf5c9ce222432006f1d93806187cd9/bing_bert/deepspeed_train.py#L79

I believe the Distributed sampler is needed to prevent the GPUS from getting duplicate samples from the generator. From looking at the Pytorch documentation, I think this works with torch.distributed.init_process_group(backend="nccl") to make sure that each copy of the model on each GPU is getting different data. The line for this code is here https://github.com/microsoft/DeepSpeedExamples/blob/fd6fb5148ccf5c9ce222432006f1d93806187cd9/bing_bert/deepspeed_train.py#L429

One question I have so far is that: shouldn't torch.distributed.init_process_group(backend="nccl") be replaced with deepspeed.init_distributed() ? Which seems what the documentation is saying

If you already have a distributed environment setup, you’d need to replace: torch.distributed.init_process_group(...) with: deepspeed.init_distributed() The default is to use the NCCL backend, which DeepSpeed has been thoroughly tested with, but you can also override the default. But if you don’t need the distributed environment setup until after deepspeed.initialize() you don’t have to use this function, as DeepSpeed will automatically initialize the distributed environment during its initialize. Regardless, you will need to remove torch.distributed.init_process_group if you already had it in place.

https://www.deepspeed.ai/getting-started/#writing-deepspeed-models

From reading this part of the documentation, to me it also reads like neither of them are needed for this code, and Deepspeed will already set up the distributed environment.

I also looked at the BingSquad example where deepspeed.init_distributed(dist_backend='nccl') was used instead.
https://github.com/microsoft/DeepSpeedExamples/blob/master/BingBertSquad/nvidia_run_squad_deepspeed.py#L745

I looked through the code of both, but I am still unable to figure out when to use torch.distributed.init_process_group(...), or deepspeed.init_distributed(dist_backend='nccl'), or neither.

As for IterableDataset, it is unable to use any type of Sampler ; if you try to put a Sampler and IterableDataset object into DataLoader you get an error stating the incompatibility.

So as of right now I am just using the dataloader with just the IterableDataset object.

train_generator = torch.utils.data.DataLoader(
    instance_IterableDataset, batch_size = None
)

Though my intuition is that this will not work. From looking at the processes Deepspeed creates, it seems to me that Deepspeed is creating a copy of the train code, one for each GPU:

santosh  32340 31761  3 04:19 pts/0    00:00:01 /opt/conda/bin/python -u kepler/deepspeed_train.py --local_rank=3
santosh  32342 31760  3 04:19 pts/0    00:00:01 /opt/conda/bin/python -u kepler/deepspeed_train.py --local_rank=2
santosh  32345 31759  3 04:19 pts/0    00:00:01 /opt/conda/bin/python -u kepler/deepspeed_train.py --local_rank=1
santosh  32353 31758  3 04:19 pts/0    00:00:01 /opt/conda/bin/python -u kepler/deepspeed_train.py --local_rank=0

So to me, it looks like just using a train generator with IterableDataset will have each GPU go through the entire dataset.

I tried just following the rest of the bing example:

    for step, dataset_type in enumerate(tqdm(dataset_picker, smoothing=1)):
        try:
            if args.config['training']['async_worker']:
                batch = worker.get()
            else:
                batch = next(dataloaders[dataset_type])

            batch = tuple(t.to(args.device) for t in batch)  # Move to GPU

            # Calculate forward pass
            loss = model.network(batch)
.
.
.
            model.network.backward(loss)

            if model.network.is_gradient_accumulation_boundary():
.
.
.
            else:
                # Call DeepSpeed engine step on micro steps
                model.network.step()

https://github.com/microsoft/DeepSpeedExamples/blob/fd6fb5148ccf5c9ce222432006f1d93806187cd9/bing_bert/deepspeed_train.py#L211

When I try with my code, using IterableDataset and neither deepspeed.init_distributed(dist_backend='nccl'), or torch.distributed.init_process_group(backend="nccl"); or I use IterableDataset with deepspeed.init_distributed(dist_backend='nccl'), I get these errors

RuntimeError: arguments are located on different GPUs at /opt/conda/conda-bld/pytorch_1595629403081/work/aten/src/THC/generic/THCTensorIndex.cu:403

and

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0!

So, it seems that I need to incorporate something else. From reading this [ https://discuss.pytorch.org/t/using-iterabledataset-with-distributeddataparallel/92589 ], perhaps DistributedDataParallel, but seems that Deepspeed is doing DistributedDataParallel's functionality already.

Here is the full error I get

Traceback (most recent call last):
  File "kepler/deepspeed_train.py", line 164, in <module>
    train(args)
  File "kepler/deepspeed_train.py", line 139, in train
    loss = model_engine(model_batch)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 928, in forward
    loss = self.module(*inputs, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/santosh/natural_language_processing/kepler/model/pipeline_parallel.py", line 334, in forward
    loss = self.pipeline(batch)
  File "cytoolz/functoolz.pyx", line 505, in cytoolz.functoolz.Compose.__call__
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/container.py", line 117, in forward
    input = module(input)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/santosh/natural_language_processing/kepler/model/pipeline_parallel.py", line 172, in forward
    words_embeddings = self.word_embeddings(input_ids.long())
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/sparse.py", line 126, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/functional.py", line 1814, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: arguments are located on different GPUs at /opt/conda/conda-bld/pytorch_1595629403081/work/aten/src/THC/generic/THCTensorIndex.cu:403
0it [00:47, ?it/s]
Traceback (most recent call last):
  File "kepler/deepspeed_train.py", line 164, in <module>
    train(args)
  File "kepler/deepspeed_train.py", line 139, in train
    loss = model_engine(model_batch)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 928, in forward
    loss = self.module(*inputs, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/santosh/natural_language_processing/kepler/model/pipeline_parallel.py", line 334, in forward
    loss = self.pipeline(batch)
  File "cytoolz/functoolz.pyx", line 505, in cytoolz.functoolz.Compose.__call__
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/container.py", line 117, in forward
    input = module(input)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/santosh/natural_language_processing/kepler/model/pipeline_parallel.py", line 172, in forward
    words_embeddings = self.word_embeddings(input_ids.long())
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/sparse.py", line 126, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/functional.py", line 1814, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: arguments are located on different GPUs at /opt/conda/conda-bld/pytorch_1595629403081/work/aten/src/THC/generic/THCTensorIndex.cu:403
0it [00:49, ?it/s]
Traceback (most recent call last):
  File "kepler/deepspeed_train.py", line 164, in <module>
    train(args)
  File "kepler/deepspeed_train.py", line 139, in train
    loss = model_engine(model_batch)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 928, in forward
    loss = self.module(*inputs, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/santosh/natural_language_processing/kepler/model/pipeline_parallel.py", line 334, in forward
    loss = self.pipeline(batch)
  File "cytoolz/functoolz.pyx", line 505, in cytoolz.functoolz.Compose.__call__
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/container.py", line 117, in forward
    input = module(input)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/santosh/natural_language_processing/kepler/model/pipeline_parallel.py", line 192, in forward
    batch["hidden_states"], batch["attention_mask"])
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/santosh/natural_language_processing/kepler/model/modeling.py", line 420, in forward
    self_output = self.self(input_tensor, attention_mask)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/santosh/natural_language_processing/kepler/model/modeling.py", line 371, in forward
    mixed_query_layer = self.query(hidden_states)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 91, in forward
    return F.linear(input, self.weight, self.bias)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/functional.py", line 1678, in linear
    output += bias
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0!
Killing subprocess 2993
Killing subprocess 2994
Killing subprocess 2995
Killing subprocess 2996
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/opt/conda/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/opt/conda/lib/python3.7/site-packages/deepspeed/launcher/launch.py", line 171, in <module>
    main()
  File "/opt/conda/lib/python3.7/site-packages/deepspeed/launcher/launch.py", line 161, in main
    sigkill_handler(signal.SIGTERM, None)  # not coming back
  File "/opt/conda/lib/python3.7/site-packages/deepspeed/launcher/launch.py", line 139, in sigkill_handler
    raise subprocess.CalledProcessError(returncode=last_return_code, cmd=cmd)
subprocess.CalledProcessError: Command '['/opt/conda/bin/python', '-u', 'kepler/deepspeed_train.py', '--local_rank=3']' returned non-zero exit status 1.

If I use IterableDataset withtorch.distributed.init_process_group(backend="nccl"), I get a slightly different error

Traceback (most recent call last):
  File "kepler/deepspeed_train.py", line 164, in <module>
    train(args)
  File "kepler/deepspeed_train.py", line 139, in train
    loss = model_engine(model_batch)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 928, in forward
    loss = self.module(*inputs, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/santosh/natural_language_processing/kepler/model/pipeline_parallel.py", line 334, in forward
    loss = self.pipeline(batch)
  File "cytoolz/functoolz.pyx", line 505, in cytoolz.functoolz.Compose.__call__
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/container.py", line 117, in forward
    input = module(input)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/santosh/natural_language_processing/kepler/model/pipeline_parallel.py", line 172, in forward
    words_embeddings = self.word_embeddings(input_ids.long())
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/sparse.py", line 126, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/functional.py", line 1814, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: arguments are located on different GPUs at /opt/conda/conda-bld/pytorch_1595629403081/work/aten/src/THC/generic/THCTensorIndex.cu:403
0it [00:48, ?it/s]
Traceback (most recent call last):
  File "kepler/deepspeed_train.py", line 164, in <module>
    train(args)
  File "kepler/deepspeed_train.py", line 139, in train
    loss = model_engine(model_batch)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 928, in forward
    loss = self.module(*inputs, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/santosh/natural_language_processing/kepler/model/pipeline_parallel.py", line 334, in forward
    loss = self.pipeline(batch)
  File "cytoolz/functoolz.pyx", line 505, in cytoolz.functoolz.Compose.__call__
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/container.py", line 117, in forward
    input = module(input)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/santosh/natural_language_processing/kepler/model/pipeline_parallel.py", line 172, in forward
    words_embeddings = self.word_embeddings(input_ids.long())
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/sparse.py", line 126, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/functional.py", line 1814, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: arguments are located on different GPUs at /opt/conda/conda-bld/pytorch_1595629403081/work/aten/src/THC/generic/THCTensorIndex.cu:403
0it [00:49, ?it/s]
Traceback (most recent call last):
  File "kepler/deepspeed_train.py", line 164, in <module>
    train(args)
  File "kepler/deepspeed_train.py", line 139, in train
    loss = model_engine(model_batch)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 928, in forward
    loss = self.module(*inputs, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/santosh/natural_language_processing/kepler/model/pipeline_parallel.py", line 334, in forward
    loss = self.pipeline(batch)
  File "cytoolz/functoolz.pyx", line 505, in cytoolz.functoolz.Compose.__call__
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/container.py", line 117, in forward
    input = module(input)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/santosh/natural_language_processing/kepler/model/pipeline_parallel.py", line 172, in forward
    words_embeddings = self.word_embeddings(input_ids.long())
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/sparse.py", line 126, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/functional.py", line 1814, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: arguments are located on different GPUs at /opt/conda/conda-bld/pytorch_1595629403081/work/aten/src/THC/generic/THCTensorIndex.cu:403
Killing subprocess 3951
Killing subprocess 3952
Killing subprocess 3953
Killing subprocess 3954
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/opt/conda/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/opt/conda/lib/python3.7/site-packages/deepspeed/launcher/launch.py", line 171, in <module>
    main()
  File "/opt/conda/lib/python3.7/site-packages/deepspeed/launcher/launch.py", line 161, in main
    sigkill_handler(signal.SIGTERM, None)  # not coming back
  File "/opt/conda/lib/python3.7/site-packages/deepspeed/launcher/launch.py", line 139, in sigkill_handler
    raise subprocess.CalledProcessError(returncode=last_return_code, cmd=cmd)
subprocess.CalledProcessError: Command '['/opt/conda/bin/python', '-u', 'kepler/deepspeed_train.py', '--local_rank=3']' returned non-zero exit status 1.

And here is the full output

lr is 0.00003
seed is 12345
master port is 29500
dropout is 0.1
deepspeed --num_nodes 1 --num_gpus 4 --master_port=29500 --hostfile /dev/null kepler/deepspeed_train.py
[2021-04-29 09:36:29,568] [WARNING] [runner.py:117:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only.
[2021-04-29 09:36:29,645] [INFO] [runner.py:355:main] cmd = /opt/conda/bin/python -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMCwgMSwgMiwgM119 --master_addr=127.0.0.1 --master_port=29500 kepler/deepspeed_train.py
[2021-04-29 09:36:30,421] [INFO] [launch.py:73:main] 0 NCCL_VERSION 2.4.8
[2021-04-29 09:36:30,421] [INFO] [launch.py:80:main] WORLD INFO DICT: {'localhost': [0, 1, 2, 3]}
[2021-04-29 09:36:30,421] [INFO] [launch.py:89:main] nnodes=1, num_local_procs=4, node_rank=0
[2021-04-29 09:36:30,421] [INFO] [launch.py:101:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0, 1, 2, 3]})
[2021-04-29 09:36:30,421] [INFO] [launch.py:102:main] dist_world_size=4
[2021-04-29 09:36:30,421] [INFO] [launch.py:105:main] Setting CUDA_VISIBLE_DEVICES=0,1,2,3
2021-04-29 09:36:32.095447: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
2021-04-29 09:36:32.095473: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2021-04-29 09:36:32.105691: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
2021-04-29 09:36:32.105691: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
2021-04-29 09:36:32.105715: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2021-04-29 09:36:32.105718: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2021-04-29 09:36:32.130134: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
2021-04-29 09:36:32.130160: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
[2021-04-29 09:36:37,689] [INFO] [logging.py:60:log_dist] [Rank -1] DeepSpeed info: version=0.3.15, git-hash=unknown, git-branch=unknown
[2021-04-29 09:36:37,689] [INFO] [distributed.py:47:init_distributed] Initializing torch distributed with backend: nccl
[2021-04-29 09:36:37,838] [INFO] [logging.py:60:log_dist] [Rank -1] DeepSpeed info: version=0.3.15, git-hash=unknown, git-branch=unknown
[2021-04-29 09:36:37,838] [INFO] [distributed.py:47:init_distributed] Initializing torch distributed with backend: nccl
[2021-04-29 09:36:37,866] [INFO] [logging.py:60:log_dist] [Rank -1] DeepSpeed info: version=0.3.15, git-hash=unknown, git-branch=unknown
[2021-04-29 09:36:37,867] [INFO] [distributed.py:47:init_distributed] Initializing torch distributed with backend: nccl
[2021-04-29 09:36:37,878] [INFO] [logging.py:60:log_dist] [Rank -1] DeepSpeed info: version=0.3.15, git-hash=unknown, git-branch=unknown
[2021-04-29 09:36:37,878] [INFO] [distributed.py:47:init_distributed] Initializing torch distributed with backend: nccl
[2021-04-29 09:36:37,900] [INFO] [engine.py:80:_initialize_parameter_parallel_groups] data_parallel_size: 4, parameter_parallel_size: 4
[2021-04-29 09:36:39,606] [INFO] [engine.py:80:_initialize_parameter_parallel_groups] data_parallel_size: 4, parameter_parallel_size: 4
[2021-04-29 09:36:39,651] [INFO] [engine.py:80:_initialize_parameter_parallel_groups] data_parallel_size: 4, parameter_parallel_size: 4
[2021-04-29 09:36:39,663] [INFO] [engine.py:80:_initialize_parameter_parallel_groups] data_parallel_size: 4, parameter_parallel_size: 4
Using /.cache/torch_extensions as PyTorch extensions root...
Using /.cache/torch_extensions as PyTorch extensions root...
Using /.cache/torch_extensions as PyTorch extensions root...
Using /.cache/torch_extensions as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /.cache/torch_extensions/fused_adam/build.ninja...
Building extension module fused_adam...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module fused_adam...
Time to load fused_adam op: 0.4849116802215576 seconds
[2021-04-29 09:36:40,943] [INFO] [engine.py:615:_configure_optimizer] Using DeepSpeed Optimizer param name adam as basic optimizer
[2021-04-29 09:36:40,943] [INFO] [engine.py:619:_configure_optimizer] DeepSpeed Basic Optimizer = FusedAdam
[2021-04-29 09:36:40,943] [INFO] [logging.py:60:log_dist] [Rank 0] Creating fp16 optimizer with dynamic loss scale
[2021-04-29 09:36:40,957] [INFO] [logging.py:60:log_dist] [Rank 0] DeepSpeed Final Optimizer = adam
[2021-04-29 09:36:40,957] [INFO] [engine.py:455:_configure_lr_scheduler] DeepSpeed using client LR scheduler
[2021-04-29 09:36:40,957] [INFO] [logging.py:60:log_dist] [Rank 0] DeepSpeed LR Scheduler = None
[2021-04-29 09:36:40,957] [INFO] [logging.py:60:log_dist] [Rank 0] step=0, skipped=0, lr=[5e-10, 5e-10], mom=[(0.9, 0.98), (0.9, 0.98)]
[2021-04-29 09:36:40,957] [INFO] [config.py:741:print] DeepSpeedEngine configuration:
[2021-04-29 09:36:40,958] [INFO] [config.py:745:print]   activation_checkpointing_config  {
    "partition_activations": false, 
    "contiguous_memory_optimization": false, 
    "cpu_checkpointing": false, 
    "number_checkpoints": null, 
    "synchronize_checkpoint_boundary": false, 
    "profile": false
}
[2021-04-29 09:36:40,958] [INFO] [config.py:745:print]   aio_config ................... {'block_size': 1048576, 'queue_depth': 8, 'thread_count': 1, 'single_submit': False, 'overlap_events': True}
[2021-04-29 09:36:40,958] [INFO] [config.py:745:print]   allreduce_always_fp32 ........ False
[2021-04-29 09:36:40,958] [INFO] [config.py:745:print]   amp_enabled .................. False
[2021-04-29 09:36:40,958] [INFO] [config.py:745:print]   amp_params ................... False
[2021-04-29 09:36:40,958] [INFO] [config.py:745:print]   checkpoint_tag_validation_enabled  True
[2021-04-29 09:36:40,958] [INFO] [config.py:745:print]   checkpoint_tag_validation_fail  False
[2021-04-29 09:36:40,958] [INFO] [config.py:745:print]   disable_allgather ............ False
[2021-04-29 09:36:40,958] [INFO] [config.py:745:print]   dump_state ................... False
[2021-04-29 09:36:40,958] [INFO] [config.py:745:print]   dynamic_loss_scale_args ...... None
[2021-04-29 09:36:40,958] [INFO] [config.py:745:print]   elasticity_enabled ........... False
[2021-04-29 09:36:40,958] [INFO] [config.py:745:print]   flops_profiler_config ........ {
    "enabled": false, 
    "profile_step": 1, 
    "module_depth": -1, 
    "top_modules": 3, 
    "detailed": true
}
[2021-04-29 09:36:40,958] [INFO] [config.py:745:print]   fp16_enabled ................. True
[2021-04-29 09:36:40,958] [INFO] [config.py:745:print]   global_rank .................. 0
[2021-04-29 09:36:40,958] [INFO] [config.py:745:print]   gradient_accumulation_steps .. 1
[2021-04-29 09:36:40,958] [INFO] [config.py:745:print]   gradient_clipping ............ 1.0
[2021-04-29 09:36:40,958] [INFO] [config.py:745:print]   gradient_predivide_factor .... 1.0
[2021-04-29 09:36:40,958] [INFO] [config.py:745:print]   initial_dynamic_scale ........ 4294967296
[2021-04-29 09:36:40,958] [INFO] [config.py:745:print]   loss_scale ................... 0
[2021-04-29 09:36:40,958] [INFO] [config.py:745:print]   memory_breakdown ............. False
[2021-04-29 09:36:40,958] [INFO] [config.py:745:print]   optimizer_legacy_fusion ...... False
[2021-04-29 09:36:40,958] [INFO] [config.py:745:print]   optimizer_name ............... adam
[2021-04-29 09:36:40,958] [INFO] [config.py:745:print]   optimizer_params ............. {'lr': 3e-05}
[2021-04-29 09:36:40,958] [INFO] [config.py:745:print]   pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0}
[2021-04-29 09:36:40,958] [INFO] [config.py:745:print]   pld_enabled .................. False
[2021-04-29 09:36:40,958] [INFO] [config.py:745:print]   pld_params ................... False
[2021-04-29 09:36:40,958] [INFO] [config.py:745:print]   prescale_gradients ........... False
[2021-04-29 09:36:40,959] [INFO] [config.py:745:print]   scheduler_name ............... None
[2021-04-29 09:36:40,959] [INFO] [config.py:745:print]   scheduler_params ............. None
[2021-04-29 09:36:40,959] [INFO] [config.py:745:print]   sparse_attention ............. None
[2021-04-29 09:36:40,959] [INFO] [config.py:745:print]   sparse_gradients_enabled ..... False
[2021-04-29 09:36:40,959] [INFO] [config.py:745:print]   steps_per_print .............. 10
[2021-04-29 09:36:40,959] [INFO] [config.py:745:print]   tensorboard_enabled .......... False
[2021-04-29 09:36:40,959] [INFO] [config.py:745:print]   tensorboard_job_name ......... DeepSpeedJobName
[2021-04-29 09:36:40,959] [INFO] [config.py:745:print]   tensorboard_output_path ...... 
[2021-04-29 09:36:40,959] [INFO] [config.py:745:print]   train_batch_size ............. 8
[2021-04-29 09:36:40,959] [INFO] [config.py:745:print]   train_micro_batch_size_per_gpu  2
[2021-04-29 09:36:40,959] [INFO] [config.py:745:print]   wall_clock_breakdown ......... False
[2021-04-29 09:36:40,959] [INFO] [config.py:745:print]   world_size ................... 4
[2021-04-29 09:36:40,959] [INFO] [config.py:745:print]   zero_allow_untested_optimizer  False
[2021-04-29 09:36:40,959] [INFO] [config.py:745:print]   zero_config .................. {
    "stage": 0, 
    "contiguous_gradients": false, 
    "reduce_scatter": true, 
    "reduce_bucket_size": 5.000000e+08, 
    "allgather_partitions": true, 
    "allgather_bucket_size": 5.000000e+08, 
    "overlap_comm": false, 
    "load_from_fp32_weights": true, 
    "elastic_checkpoint": true, 
    "offload_param": null, 
    "offload_optimizer": null, 
    "sub_group_size": 1.000000e+12, 
    "prefetch_bucket_size": 5.000000e+07, 
    "param_persistence_threshold": 1.000000e+05, 
    "max_live_parameters": 1.000000e+09, 
    "max_reuse_distance": 1.000000e+09, 
    "gather_fp16_weights_on_model_save": false
}
[2021-04-29 09:36:40,959] [INFO] [config.py:745:print]   zero_enabled ................. False
[2021-04-29 09:36:40,959] [INFO] [config.py:745:print]   zero_optimization_stage ...... 0
[2021-04-29 09:36:40,959] [INFO] [config.py:752:print]   json = {
    "train_batch_size": 8, 
    "train_micro_batch_size_per_gpu": 2, 
    "steps_per_print": 10, 
    "optimizer": {
        "type": "Adam", 
        "params": {
            "lr": 3e-05
        }
    }, 
    "gradient_clipping": 1.0, 
    "fp16": {
        "enabled": true
    }
}
Using /.cache/torch_extensions as PyTorch extensions root...
Loading extension module fused_adam...
Time to load fused_adam op: 0.5048720836639404 seconds
Loading extension module fused_adam...
Time to load fused_adam op: 0.5043542385101318 seconds
Loading extension module fused_adam...
Time to load fused_adam op: 0.5045933723449707 seconds
Using /.cache/torch_extensions as PyTorch extensions root...
Using /.cache/torch_extensions as PyTorch extensions root...
Using /.cache/torch_extensions as PyTorch extensions root...
Emitting ninja build file /.cache/torch_extensions/utils/build.ninja...
Building extension module utils...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module utils...
Time to load utils op: 0.42908811569213867 seconds
hi
0it [00:00, ?it/s]Loading extension module utils...
Time to load utils op: 0.40368103981018066 seconds
hi
0it [00:00, ?it/s]Loading extension module utils...
Time to load utils op: 0.403658390045166 seconds
hi
Loading extension module utils...
0it [00:00, ?it/s]Time to load utils op: 0.4037015438079834 seconds
hi
0it [00:47, ?it/s]
Traceback (most recent call last):
  File "kepler/deepspeed_train.py", line 164, in <module>
    train(args)
  File "kepler/deepspeed_train.py", line 139, in train
    loss = model_engine(model_batch)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 928, in forward
    loss = self.module(*inputs, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/santosh/natural_language_processing/kepler/model/pipeline_parallel.py", line 334, in forward
    loss = self.pipeline(batch)
  File "cytoolz/functoolz.pyx", line 505, in cytoolz.functoolz.Compose.__call__
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/container.py", line 117, in forward
    input = module(input)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/santosh/natural_language_processing/kepler/model/pipeline_parallel.py", line 172, in forward
    words_embeddings = self.word_embeddings(input_ids.long())
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/sparse.py", line 126, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/functional.py", line 1814, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: arguments are located on different GPUs at /opt/conda/conda-bld/pytorch_1595629403081/work/aten/src/THC/generic/THCTensorIndex.cu:403
0it [00:47, ?it/s]
Traceback (most recent call last):
  File "kepler/deepspeed_train.py", line 164, in <module>
    train(args)
  File "kepler/deepspeed_train.py", line 139, in train
    loss = model_engine(model_batch)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 928, in forward
    loss = self.module(*inputs, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/santosh/natural_language_processing/kepler/model/pipeline_parallel.py", line 334, in forward
    loss = self.pipeline(batch)
  File "cytoolz/functoolz.pyx", line 505, in cytoolz.functoolz.Compose.__call__
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/container.py", line 117, in forward
    input = module(input)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/santosh/natural_language_processing/kepler/model/pipeline_parallel.py", line 172, in forward
    words_embeddings = self.word_embeddings(input_ids.long())
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/sparse.py", line 126, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/functional.py", line 1814, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: arguments are located on different GPUs at /opt/conda/conda-bld/pytorch_1595629403081/work/aten/src/THC/generic/THCTensorIndex.cu:403
0it [00:49, ?it/s]
Traceback (most recent call last):
  File "kepler/deepspeed_train.py", line 164, in <module>
    train(args)
  File "kepler/deepspeed_train.py", line 139, in train
    loss = model_engine(model_batch)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 928, in forward
    loss = self.module(*inputs, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/santosh/natural_language_processing/kepler/model/pipeline_parallel.py", line 334, in forward
    loss = self.pipeline(batch)
  File "cytoolz/functoolz.pyx", line 505, in cytoolz.functoolz.Compose.__call__
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/container.py", line 117, in forward
    input = module(input)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/santosh/natural_language_processing/kepler/model/pipeline_parallel.py", line 192, in forward
    batch["hidden_states"], batch["attention_mask"])
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/santosh/natural_language_processing/kepler/model/modeling.py", line 420, in forward
    self_output = self.self(input_tensor, attention_mask)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/santosh/natural_language_processing/kepler/model/modeling.py", line 371, in forward
    mixed_query_layer = self.query(hidden_states)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 91, in forward
    return F.linear(input, self.weight, self.bias)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/functional.py", line 1678, in linear
    output += bias
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0!
Killing subprocess 2993
Killing subprocess 2994
Killing subprocess 2995
Killing subprocess 2996
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/opt/conda/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/opt/conda/lib/python3.7/site-packages/deepspeed/launcher/launch.py", line 171, in <module>
    main()
  File "/opt/conda/lib/python3.7/site-packages/deepspeed/launcher/launch.py", line 161, in main
    sigkill_handler(signal.SIGTERM, None)  # not coming back
  File "/opt/conda/lib/python3.7/site-packages/deepspeed/launcher/launch.py", line 139, in sigkill_handler
    raise subprocess.CalledProcessError(returncode=last_return_code, cmd=cmd)
subprocess.CalledProcessError: Command '['/opt/conda/bin/python', '-u', 'kepler/deepspeed_train.py', '--local_rank=3']' returned non-zero exit status 1.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with kepler/deepspeed_train.py lines 139 and 164 from the traceback, then inspect model/pipeline_parallel.py lines 172 and 334. Compare the IterableDataset, DataLoader, and distributed initialization against the linked DeepSpeedExamples files and PyTorch discussion. Done means identifying the cause of the cross-GPU errors and documenting a reproducible, supported setup.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
data-engineering, distributed-systems, machine-learning
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.