codertimo / codertimo/BERT-pytorch
Clarification on Padding Process in BERT Model Construction
- Dominant language
- Python
- Stars
- 6.5k
- Forks
- 1.3k
- PR merge metrics
- No merged PRs in 30d
Description
Hi there,
In my endeavor to construct a BERT model from the ground up for the purpose of gaining hands-on experience and a comprehensive understanding of the model, I have encountered a point of confusion regarding the padding process.
According to my interpretation of the `dataset.py` file, the standard procedure involves concatenating two sentences and adding pad tokens at the end of the combined sequence, similar to typical padding practices in other tasks. Here's an example:
```
Sent_A= The cat is walking (Len is 4)
Sent_B= The dog is barking at the tree (Len is 7)
Maximum sequence length: 15
Sequence: The cat is walking The dog is barking at the tree [PAD] [PAD] [PAD] [PAD]
Segment: 0 0 0 0 1 1 1 1 1 1 1 0 0 0 0
```
However, in alternative implementations [1](https://neptune.ai/blog/how-to-code-bert-using-pytorch-tutorial) [2](https://coaxsoft.com/blog/building-bert-with-pytorch-from-scratch), I've observed padding aimed at equalizing the lengths of the two sentences, resulting in a structure like this:
```
Sent_A= The cat is walking (Len is 4)
Sent_B= The dog is barking at the tree (Len is 7)
Maximum sequence length: 15
Sequence: The cat is walking [PAD] [PAD] [PAD] The dog is barking at the tree [PAD]
Segment: 0 0 0 0 0 0 0 1 1 1 1 1 1 1 0
```
This discrepancy has left me in a state of uncertainty—whether my understanding from the `dataset.py` file is accurate, or if the alternative implementations I've come across have a different rationale behind their padding approach. Could you kindly provide clarification on this matter?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.