mlfoundations / mlfoundations/datacomp
Tried evaluate the model on a local network only machine
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 792
- Forks
- 65
- PR merge metrics
- No merged PRs in 30d
Description
Well, I first use
python download_evalsets.py $download_dir
to download all the necessary datasets on an internet-accessible machine and then migrate the data to my machine with limited internet access.
All the other evaluation went well but the retrieval datasets , which use hf_cache/ directory instead.
The error goes like this :
>>> datasets.load_dataset("nlphuji/flickr_1k_test_image_text_retrieval",split="test", cache_dir=os.path.join("/mnt/data/datacom2023/evaluate_datasets", "hf_cache"))Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/root/anaconda3/envs/datacomp/lib/python3.10/site-packages/datasets/load.py", line 2129, in load_dataset
builder_instance = load_dataset_builder(
File "/root/anaconda3/envs/datacomp/lib/python3.10/site-packages/datasets/load.py", line 1815, in load_dataset_builder
dataset_module = dataset_module_factory(
File "/root/anaconda3/envs/datacomp/lib/python3.10/site-packages/datasets/load.py", line 1512, in dataset_module_factory
raise e1 from None
File "/root/anaconda3/envs/datacomp/lib/python3.10/site-packages/datasets/load.py", line 1468, in dataset_module_factory
raise ConnectionError(f"Couldn't reach '{path}' on the Hub ({type(e).__name__})")
ConnectionError: Couldn't reach 'nlphuji/flickr_1k_test_image_text_retrieval' on the Hub (ConnectTimeout)
Seems like the huggingface datasets module is still trying to connect to the internet. Is there any trick I can play to skip the connection to huggingface? The evaluation command :
python evaluate.py --train_output_dir /mnt/data/datacomp2023/train_output/basic_train --data_dir /mnt/data/datacomp2023/evaluate_datasets
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading download_evalsets.py and evaluate.py, then inspect how the retrieval datasets use the hf_cache directory and call datasets.load_dataset. Reproduce the command on the limited-access machine and determine what offline behavior is needed; done means retrieval evaluation completes without contacting the Hugging Face Hub.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- huggingface, python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100