facebookresearch / facebookresearch/blt

Question about Encoder Hash n-gram Embeddings

Open
#29 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.1k
Forks
196
PR merge metrics
No merged PRs in 30d

Description

The paper said that the window size is selected from values ​​from 3 to 8. But in the code and in debug.yaml there is simply 4 and the enumeration is based on the primal number.

https://github.com/facebookresearch/blt/blob/main/bytelatent/model/blt.py#L756 and corresponding part 3.2.1 Encoder Hash n-gram Embeddings in paper

Params in debug.yaml:
encoder_hash_byte_group_nb_functions: 3
encoder_hash_byte_group_size: [4]

Am I right according to the paper should be:
encoder_hash_byte_group_nb_functions: 1
encoder_hash_byte_group_size: [3,4,5,6,7,8]

Is this a mistake because you did not share the correct config or does it train better this way?

Contributor guide

Open the contributing guide

Research direction

Compare the Encoder Hash n-gram Embeddings description in section 3.2.1 of the paper with the implementation at blt.py#L756 and the encoder_hash_byte_group settings in debug.yaml. Determine whether the single size value and three hash functions are intentional or an incorrect shared configuration, then document or correct the discrepancy and verify the resulting configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.