CarperAI / CarperAI/trlx

Add tensor padding options to improve hardware utilization

Open
#284 8 comments 2 reactions 0 assignees View on GitHub
contributions welcome feature request good first issue
Dominant language
Python
Stars
4.8k
Forks
487
PR merge metrics
No merged PRs in 30d

Description

### 🚀 The feature, motivation, and pitch

Recent [discussion](https://twitter.com/karpathy/status/1621578354024677377) on Twitter highlighted the importance of tensor padding to improve hardware util. NeMo seems to already support GPU-friendly vocab size padding from discussions with @cat-state but we should also consider adding an optional [`pad_to_multiple_of`](https://huggingface.co/docs/transformers/main_classes/tokenizer#transformers.PreTrainedTokenizer.__call__.pad_to_multiple_of) arg to our tokenizer calls with the accelerate/transformers backend to satisfy [Tensor Core requirements](https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html#requirements-tc).

![image](https://user-images.githubusercontent.com/41410219/217096562-32589fb5-817d-40fc-bd89-ccc909c40516.png)

The implementer should also provide system plots to display any improvements/findings.

### Alternatives

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.