Lightning-AI / Lightning-AI/pytorch-lightning

Profiling with DLProf Returns Error "No pytorch ops detected"

Open
#18,311 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug profiler ver: 2.0.x
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

### Bug description

I am trying to run profiling using DLProf to get system metrics, but everytime training finishes, DLProf returns an error saying no pytorch ops detected. I also used PyTorch Profiler and set the `emit_nvtx` to True. I don't know what I'm missing here.

### What version are you seeing the problem on?

v2.0

### How to reproduce the bug

```python
args = get_opts()
system = NeRF_pl(args) # PL module

logger = pl.loggers.TensorBoardLogger(save_dir=args.logs_dir,
name=args.exp_name,
default_hp_metric=False)

profiler = PyTorchProfiler(
on_trace_ready=torch.profiler.tensorboard_trace_handler(args.logs_dir),
schedule=torch.profiler.schedule(skip_first=10, wait=1, warmup=1, active=20),
trace_memory=False,
profile_memory=True,
emit_nvtx=True,
# export_to_chrome=True,
use_cpu=True,
# use_cuda=True
)

ckpt_callback = pl.callbacks.ModelCheckpoint(dirpath="{}/{}".format(args.ckpts_dir, args.exp_name),
filename="{epoch:d}",
monitor="val/psnr",
mode="max",
save_top_k=-1,
# every_n_epochs=args.save_every_n_epochs)
every_n_epochs=1)

trainer = pl.Trainer(max_epochs=args.max_epochs,
logger=logger,
callbacks=[ckpt_callback],
devices=3,
# deterministic=True, # RuntimeError: cumsum_cuda_kernel does not have a deterministic implementation, but you set 'torch.use_deterministic_algorithms(True)'.
benchmark=True,
# weights_summary=None, # pass a ModelSummary callback with max_depth instead
num_sanity_val_steps=2,
check_val_every_n_epoch=1,
profiler=profiler)

trainer.fit(system, ckpt_path=args.ckpt_path)

```bash
dlprof --mode pytorch python3 main.py
```

### Error messages and logs

```
[DLProf-08:20:24] Error Occurred:
[DLProf-08:20:24] No pytorch ops detected. Make sure that you add nvidia_dlprof_pytorch_nvtx to your source code. Refer to: https://docs.nvidia.com/deeplearning/frameworks/dlprof-user-guide/index.html for more information.
```

### Environment

Current environment

```
#- Lightning Component (e.g. Trainer, LightningModule, LightningApp, LightningWork, LightningFlow):
#- PyTorch Lightning Version: 2.0.6
#- Lightning App Version: -
#- PyTorch Version: 2.0.1
#- Python version: 3.8
#- OS: Ubuntu 22.04
#- CUDA/cuDNN version:
#- GPU models and configuration: RTX A6000
#- How you installed Lightning: pip
#- Running environment of LightningApp (e.g. local, cloud): -
```

### More info

_No response_

cc @carmocca @nbcsm @guotuofeng

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the provided PyTorchProfiler configuration, the trainer.fit reproduction, and the `dlprof --mode pytorch python3 main.py` command. Read the linked DLProf user guide and investigate how Lightning 2.0.6 exposes PyTorch profiler traces to DLProf. Done means the cause of the missing operations is identified and the documented reproduction either produces usable profiling data or has a confirmed compatibility limitation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.