allenai / allenai/oocmap

Support for Numpy

Abierto
#6 0 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
C
Estrellas
12
Forks
1
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

We have to store Numpy arrays in this.

For some size of tensor (larger than 4kb?), it would be a killer feature if we could lazy-load the entire tensor. That would mean that `LazyTensor` (or whatever class we use for this) keeps an open LMDB transaction, so that the pointers to the data stay alive for as long as possible. If we do this, we will need some mechanism yoink transactions from live objects when we run out of transaction slots.

Or maybe it would be better if large tensors get their own memory-mapped file. That means they are not transactional like the rest of OOCMap is. I'm not sure that's really a problem.

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.