hyperledger-labs / hyperledger-labs/aifaq
MistralAI tokenizer warning persists despite setting HF_TOKEN environment variable
Open
- Dominant language
- TypeScript
- Stars
- 64
- Forks
- 64
- Avg merge
- 6m
- Merged PRs (30d)
- 5
Description
**When running python ingest.py:**
Despite properly setting the HF_TOKEN environment variable, the following warning still appears when running the document ingestion script:
`
/Users/somnathumapathi/my_projects/aifaq/src/mvt/venv/lib/python3.13/site-packages/langchain_mistralai/embeddings.py:181: UserWarning: Could not download mistral tokenizer from Huggingface for calculating batch sizes. Set a Huggingface token via the HF_TOKEN environment variable to download the real tokenizer. Falling back to a dummy tokenizer that uses `len()`.`
Contributor guide
Assessment
This issue has not been assessed yet.