hyperledger-labs / hyperledger-labs/aifaq

MistralAI tokenizer warning persists despite setting HF_TOKEN environment variable

Open
#116 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
64
Forks
64
Avg merge
6m
Merged PRs (30d)
5

Description

Image

**When running python ingest.py:**

Despite properly setting the HF_TOKEN environment variable, the following warning still appears when running the document ingestion script:

`
/Users/somnathumapathi/my_projects/aifaq/src/mvt/venv/lib/python3.13/site-packages/langchain_mistralai/embeddings.py:181: UserWarning: Could not download mistral tokenizer from Huggingface for calculating batch sizes. Set a Huggingface token via the HF_TOKEN environment variable to download the real tokenizer. Falling back to a dummy tokenizer that uses `len()`.`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.