cdpierse / cdpierse/transformers-interpret

Token Classification Memory Issue

Open
#117 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
1.4k
Forks
99
PR merge metrics
No merged PRs in 30d

Description

Hi there, I am using `TokenClassificationExplainer` which works great with small text inputs. However, for a large 512 token input my machine runs out of memory and crashes.

This is probably because of an [unavoidable?] exponential memory issue of creating attributions for all 512 subtokens for each subtoken. Normally, the local context is most likely to contribute to the classificaiton of each token, it could be useful to reduce the number of pairs we make for each token. For example, the 512th token is less related to the 10th token, than tokens 1 - 20.

Would it be possible to add in a `context_window_size: Optional[int] = None` somewhere ([possibly here](https://github.com/cdpierse/transformers-interpret/blob/master/transformers_interpret/explainer.py#L104)), to reduce the number of pairs, so we we only calculate attributions for the `context_window_size` many tokens to the left and right.

I have tried to dig into the code to add this functionality in a PR but I don't understand all of the functions as they are missing doc strings and comments ([in here](https://github.com/cdpierse/transformers-interpret/blob/master/transformers_interpret/explainers/text/token_classification.py)).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.