facebookresearch / facebookresearch/blt
Question about Encoder Hash n-gram Embeddings
- Dominant language
- Python
- Stars
- 2.1k
- Forks
- 196
- PR merge metrics
- No merged PRs in 30d
Description
The paper said that the window size is selected from values from 3 to 8. But in the code and in debug.yaml there is simply 4 and the enumeration is based on the primal number.
https://github.com/facebookresearch/blt/blob/main/bytelatent/model/blt.py#L756 and corresponding part 3.2.1 Encoder Hash n-gram Embeddings in paper
Params in debug.yaml:
encoder_hash_byte_group_nb_functions: 3
encoder_hash_byte_group_size: [4]
Am I right according to the paper should be:
encoder_hash_byte_group_nb_functions: 1
encoder_hash_byte_group_size: [3,4,5,6,7,8]
Is this a mistake because you did not share the correct config or does it train better this way?
Contributor guide
Research direction
Compare the Encoder Hash n-gram Embeddings description in section 3.2.1 of the paper with the implementation at blt.py#L756 and the encoder_hash_byte_group settings in debug.yaml. Determine whether the single size value and three hash functions are intentional or an incorrect shared configuration, then document or correct the discrepancy and verify the resulting configuration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100