How to bring my tokenizer and set vocabulary size accordingly for training a model with loaded weights

Nobody has claimed this yet.

Hi,

I totally new in the LLM word and, first, I try training just a LLM (regular) for few files I have.

I run SentencePiece (BPE algorithm) on my few text files - that yields a tokenizer. I viewed the tokenizing table and it is just 65 rows - meaning vocabulary size = 65.

My question - how do you continue to train a model but with your own specific tokenizer and vocabulary size that is different (smaller) than the one the model was trained on.

**What are the files in python or config that I should modify ?**

**Generally speaking, at first glance I thought I would follow these steps:**
1) Load the model.
2) Change the layer (which I do not know which one yet) that corresponding to vocabulary size.
3) Load the weights into the model - hopefully I might get an warring that some wights have not been assigned du to the fact that I decrease vocabulary size to be smaller (just 65 in my case).
4) Continue training.

**I thank everybody who can guide me how to do so - and share his/her knowledge.
If there are any links or videos - please share these, too.**

**Thanks a lot**

The issue names SentencePiece, a custom tokenizer, vocabulary size, model loading, and weights, but no project files, tests, or entry points. Start by locating the tokenizer and model configuration used for training; done would be a clear, project-specific guide explaining the supported workflow for changing vocabulary size with loaded weights.

Written by the indexing model from the issue text.