pytorch / pytorch/ignite

Filter ignite stacktrace

Open
#229 4 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
4.8k
Forks
726
Avg merge
5d 21h
Merged PRs (30d)
5

Description

Should we consider filtering out the frames that come from ignite? Whenever there's a bug in my code I have to go through a lot of frames to find where ignite trace ends, and my code starts.

I think it is possible to do in Python.

E.g. I'd like to turn

/Mine/ml-mine/code.../lm/train.py in main(train, dev, nn_type, ninp, nhid, nlayers, dropout, dropouth, dropouti, dropoute, wdrop, tie_weights, bsz, bptt, max_epochs, lr, clip, adaptive, cutoffs, device, to_device, ckpt_embedding, lock_emb, kernel_size, num_levels, seed)
    238 
    239     logger.debug(f'Invoking trainer.run for {max_epochs} epochs')
--> 240     return trainer.run(data=trn_dset, max_epochs=max_epochs)

.../ignite/ignite/engine/engine.py in run(self, data, max_epochs)
    264         except BaseException as e:
    265             self._logger.error("Engine run is terminating due to exception: %s", str(e))
--> 266             self._handle_exception(e)
    267 
    268         return self.state

.../ignite/ignite/engine/engine.py in _handle_exception(self, e)
    222     def _handle_exception(self, e):
    223         if Events.EXCEPTION_RAISED in self._event_handlers:
--> 224             self._fire_event(Events.EXCEPTION_RAISED, e)
    225         else:
    226             raise e

.../ignite/ignite/engine/engine.py in _fire_event(self, event_name, *event_args)
    182             self._logger.debug("firing handlers for event %s ", event_name)
    183             for func, args, kwargs in self._event_handlers[event_name]:
--> 184                 func(self, *(event_args + args), **kwargs)
    185 
    186     def terminate(self):

.../lm/train.py in handle_exception(engine, e)
    178 
    179         else:
--> 180             raise e
    181 
    182     @validator.process_function()

.../ignite/ignite/engine/engine.py in run(self, data, max_epochs)
    251                 self.state.epoch += 1
    252                 self._fire_event(Events.EPOCH_STARTED)
--> 253                 hours, mins, secs = self._run_once_on_dataset()
    254                 self._logger.info("Epoch[%s] Complete. Time taken: %02d:%02d:%02d", self.state.epoch, hours, mins, secs)
    255                 if self.should_terminate:

.../ignite/ignite/engine/engine.py in _run_once_on_dataset(self)
    213         except BaseException as e:
    214             self._logger.error("Current run is terminating due to exception: %s", str(e))
--> 215             self._handle_exception(e)
    216 
    217         time_taken = time.time() - start_time

.../ignite/ignite/engine/engine.py in _handle_exception(self, e)
    222     def _handle_exception(self, e):
    223         if Events.EXCEPTION_RAISED in self._event_handlers:
--> 224             self._fire_event(Events.EXCEPTION_RAISED, e)
    225         else:
    226             raise e

.../ignite/ignite/engine/engine.py in _fire_event(self, event_name, *event_args)
    182             self._logger.debug("firing handlers for event %s ", event_name)
    183             for func, args, kwargs in self._event_handlers[event_name]:
--> 184                 func(self, *(event_args + args), **kwargs)
    185 
    186     def terminate(self):

.../lm/train.py in handle_exception(engine, e)
    178 
    179         else:
--> 180             raise e
    181 
    182     @validator.process_function()

.../ignite/ignite/engine/engine.py in _run_once_on_dataset(self)
    205                 self.state.iteration += 1
    206                 self._fire_event(Events.ITERATION_STARTED)
--> 207                 self.state.output = self._process_function(self, batch)
    208                 self._fire_event(Events.ITERATION_COMPLETED)
    209                 if self.should_terminate or self.should_terminate_single_epoch:

.../lm/train.py in step(engine, batch)
    107         output  = output.view(-1, output.size(2))
    108 
--> 109         loss = loss_fn(output, y)
    110         loss.backward()
    111 

into

/Mine/ml-mine/code.../lm/train.py in main(train, dev, nn_type, ninp, nhid, nlayers, dropout, dropouth, dropouti, dropoute, wdrop, tie_weights, bsz, bptt, max_epochs, lr, clip, adaptive, cutoffs, device, to_device, ckpt_embedding, lock_emb, kernel_size, num_levels, seed)
    238 
    239     logger.debug(f'Invoking trainer.run for {max_epochs} epochs')
--> 240     return trainer.run(data=trn_dset, max_epochs=max_epochs)

<PYTORCH-IGNITE STACKTRACE SKIPPED. USE `ignite.set_stacktrace(True)`>

.../lm/train.py in step(engine, batch)
    107         output  = output.view(-1, output.size(2))
    108 
--> 109         loss = loss_fn(output, y)
    110         loss.backward()
    111 

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the Python exception flow in ignite/engine/engine.py, especially run, _run_once_on_dataset, _handle_exception, and _fire_event. Use the example stacktrace to determine the desired filtering boundary, and verify that Ignite frames can be omitted while ignite.set_stacktrace(True) preserves the full trace.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
developer-experience, machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.