facebookresearch / facebookresearch/fairseq2

Add Basic building blocks for custom collators

Open
#1,335 0 comments 0 reactions 0 assignees View on GitHub
data pipeline enhancement
Dominant language
Python
Stars
1.1k
Forks
144
Avg merge
4d 1h
Merged PRs (30d)
1

Description

**Is your feature request related to a problem? Please describe:**
I have a 2 dimensional tensor I'm trying to collate, with dimension varying across two dimensions. For instance, the first example has [[1, 2], [1, 2, 3]] and the second example has [[1, 2, 3, 4]].

I found that fairseq2.data.Collator throws an exception on such tensors, and I'm trying to find a way around this without adding a lot of custom code.

**Describe the solution you would like:**
I think the Collator class should either a) support multidimensional padding or b) provide some mechanism for custom collating of specific tensors. Currently it doesn't seem to be the case, and I may need to replace the Collate class altogether to support this.

**Describe the alternatives you have considered:**
I tried using builder.map with "selector" when applying to the Collator, but I think it's not working because a) if I specify some columns like [*].text if applies collating on each sample separately and not across the tensor, and b) it may discard uncollated tensors (not sure about the latter).

**Additional Context:**
-

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.