facebookresearch / facebookresearch/sam3

Move tokenizer vocab `.gzip` into `sam3`?

Open
#170 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
11.7k
Forks
1.8k
PR merge metrics
No merged PRs in 30d

Description

Thanks for the amazing contribution!

Any chance we can move the [tokenizer vocab](https://github.com/facebookresearch/sam3/blob/main/assets/bpe_simple_vocab_16e6.txt.gz) file into the `sam3` module itself? I think this is general best practice since it's an intrinsic part of the model?

Reason being is because if one wants to add `sam3` as a submodule or `uv add git+https://github.com/facebookresearch/sam3.git` it will only install the `sam3` module and when loading `sam3` with defaults it searches for `../assets/` which is not included, thus raising an error.

Contributor guide

Open the contributing guide

Research direction

Start with assets/bpe_simple_vocab_16e6.txt.gz and the sam3 default-loading entry point that currently searches ../assets/. Check how uv add git+https://github.com/facebookresearch/sam3.git packages the module and verify that the tokenizer vocabulary is included and defaults load successfully from the installed sam3 module.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system, developer-experience
Issue type
Refactor
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.