facebookresearch / facebookresearch/fairseq2
Add Basic building blocks for custom collators
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 144
- Avg merge
- 4d 1h
- Merged PRs (30d)
- 1
Description
**Is your feature request related to a problem? Please describe:**
I have a 2 dimensional tensor I'm trying to collate, with dimension varying across two dimensions. For instance, the first example has [[1, 2], [1, 2, 3]] and the second example has [[1, 2, 3, 4]].
I found that fairseq2.data.Collator throws an exception on such tensors, and I'm trying to find a way around this without adding a lot of custom code.
**Describe the solution you would like:**
I think the Collator class should either a) support multidimensional padding or b) provide some mechanism for custom collating of specific tensors. Currently it doesn't seem to be the case, and I may need to replace the Collate class altogether to support this.
**Describe the alternatives you have considered:**
I tried using builder.map with "selector" when applying to the Collator, but I think it's not working because a) if I specify some columns like [*].text if applies collating on each sample separately and not across the tensor, and b) it may discard uncollated tensors (not sure about the latter).
**Additional Context:**
-
Contributor guide
Assessment
This issue has not been assessed yet.