huggingface / huggingface/pytorch-image-models

[FEATURE] Effective drop path.

Open
#1,836 9 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
37.1k
Forks
5.2k
Avg merge
1d 13h
Merged PRs (30d)
34

Description

**Is your feature request related to a problem? Please describe.**

While current drop path implementation in TIMM doesn't save computation resources, implementing a **true** drop path that ignores unnecessary tokens will significantly speed up training when `path drop ratio` is high (e.g. 0.3 or 0.4).

**Describe the solution you'd like**
Reference to: https://github.com/facebookresearch/dinov2/blob/c3c2683a13cde94d4d99f523cf4170384b00c34c/dinov2/layers/block.py#L110

I already implemented a modified Block that utilizes this function and it gives me a huge performance improvement. I can add it to PR #1835 if it's a good idea.

Contributor guide

Open the contributing guide

Research direction

Start by reading the referenced DINOv2 block implementation and compare it with the modified Block proposed for PR #1835. Verify that high path-drop ratios skip unnecessary token computation while preserving model behavior, then benchmark training to confirm the expected resource savings.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.