Lightning-AI / Lightning-AI/pytorch-lightning
Profiling with DLProf Returns Error "No pytorch ops detected"
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
### Bug description
I am trying to run profiling using DLProf to get system metrics, but everytime training finishes, DLProf returns an error saying no pytorch ops detected. I also used PyTorch Profiler and set the `emit_nvtx` to True. I don't know what I'm missing here.
### What version are you seeing the problem on?
v2.0
### How to reproduce the bug
```python
args = get_opts()
system = NeRF_pl(args) # PL module
logger = pl.loggers.TensorBoardLogger(save_dir=args.logs_dir,
name=args.exp_name,
default_hp_metric=False)
profiler = PyTorchProfiler(
on_trace_ready=torch.profiler.tensorboard_trace_handler(args.logs_dir),
schedule=torch.profiler.schedule(skip_first=10, wait=1, warmup=1, active=20),
trace_memory=False,
profile_memory=True,
emit_nvtx=True,
# export_to_chrome=True,
use_cpu=True,
# use_cuda=True
)
ckpt_callback = pl.callbacks.ModelCheckpoint(dirpath="{}/{}".format(args.ckpts_dir, args.exp_name),
filename="{epoch:d}",
monitor="val/psnr",
mode="max",
save_top_k=-1,
# every_n_epochs=args.save_every_n_epochs)
every_n_epochs=1)
trainer = pl.Trainer(max_epochs=args.max_epochs,
logger=logger,
callbacks=[ckpt_callback],
devices=3,
# deterministic=True, # RuntimeError: cumsum_cuda_kernel does not have a deterministic implementation, but you set 'torch.use_deterministic_algorithms(True)'.
benchmark=True,
# weights_summary=None, # pass a ModelSummary callback with max_depth instead
num_sanity_val_steps=2,
check_val_every_n_epoch=1,
profiler=profiler)
trainer.fit(system, ckpt_path=args.ckpt_path)
```bash
dlprof --mode pytorch python3 main.py
```
### Error messages and logs
```
[DLProf-08:20:24] Error Occurred:
[DLProf-08:20:24] No pytorch ops detected. Make sure that you add nvidia_dlprof_pytorch_nvtx to your source code. Refer to: https://docs.nvidia.com/deeplearning/frameworks/dlprof-user-guide/index.html for more information.
```
### Environment
Current environment
```
#- Lightning Component (e.g. Trainer, LightningModule, LightningApp, LightningWork, LightningFlow):
#- PyTorch Lightning Version: 2.0.6
#- Lightning App Version: -
#- PyTorch Version: 2.0.1
#- Python version: 3.8
#- OS: Ubuntu 22.04
#- CUDA/cuDNN version:
#- GPU models and configuration: RTX A6000
#- How you installed Lightning: pip
#- Running environment of LightningApp (e.g. local, cloud): -
```
### More info
_No response_
cc @carmocca @nbcsm @guotuofeng
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the provided PyTorchProfiler configuration, the trainer.fit reproduction, and the `dlprof --mode pytorch python3 main.py` command. Read the linked DLProf user guide and investigate how Lightning 2.0.6 exposes PyTorch profiler traces to DLProf. Done means the cause of the missing operations is identified and the documented reproduction either produces usable profiling data or has a confirmed compatibility limitation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100