allenai / allenai/ir_datasets

Lock for writing to the cache files

Abierto
#228 2 comentarios 3 reacciones 0 asignados Ver en GitHub
enhancement
Lenguaje dominante
Python
Estrellas
391
Forks
58
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

I sometimes have multiple processes or multiple machines accessing the same storage cluster that hosts the cache directory of `ir_datasets`. If multiple processes decide to download the same dataset at the same time, they start writing to the same file and eventually crash.
It would be nice if there is a locking mechanism that prevents more than one process from writing to the same file and asking other processes to wait.

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.