[Feature]: make config.json and special_tokens_map.json optional

What feature would you like to request?

Loading a tokenizer currently requires three files:

1. config.json: only used to find the pad_token_id
2. tokenizer_config.json: used to find the pad_token (not the ID)
3. special_tokens_map.json: used to find all special tokens.

out of these three tokenizer_config.json is the only one that's really needed. The other ones should be made optional.

• pad_token_id: can be retrieved by looking up the pad_token in the tokenizer's vocabulary if config.json is not present
• special_tokens_map.json: this file is no longer written by transformers in newer versions, so it is missing for newer models. Furthermore, all special tokens can be looked up in the tokenizer itself, so they don't need to be put in this map.

So what I would advocate for is:

1. Make config.json optional. Still read from it if it is present, to not break current users' expectations
2. Make special_tokens_map optional. Again, still read from it, but also derive all special tokens from the tokenizer.

This way, it becomes a lot easier to add new models to fastembed.

Is there any additional information you would like to provide?

No response

Start by locating the tokenizer-loading entry point and any existing tests covering config.json, tokenizer_config.json, and special_tokens_map.json. Verify the loader still reads those files when present, works when config.json and special_tokens_map.json are absent, and derives pad_token_id and special tokens from the tokenizer where needed.