pytorch / pytorch/ignite

Are there any ways to filter out or ignore a batch in engine?

Open
#996 8 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement help wanted
Dominant language
Python
Stars
4.8k
Forks
726
Avg merge
5d 21h
Merged PRs (30d)
5

Description

❓ Questions/Help/Support

I'm really new to the ML area and I'm trying to train a network with a dataset that when sampling batches, it can create batches with really large sizes (the sampling method is just torch's weightedSampling). Though the batch size is fixed the data size in the batch will be really large sometimes.
It seems pretty complicated to define a customized sampler, and the way I originally did was just ignore that batch when the data size in the batch is too large. Now, I'm trying to use ignite as the framework to train the network, I don't know how to do this.
I think it might relate to event_filter? But it seems to be triggered by event, so are there any ways to pass in a batch instead?

Thanks so much for your time!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue mentions Ignite's event_filter and the training engine's batch handling, but names no files or tests. Start by tracing how batches enter the engine and how event_filter is applied; determine whether skipping an oversized batch is supported without a custom sampler, and define the expected behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.