activeloopai / activeloopai/deeplake

[FEATURE] Option to disable auto commit after data ingestion

Abierto
#2,521 1 comentario 0 reacciones 1 asignado Reclamado por @FayazRahman Ver en GitHub
enhancement
Lenguaje dominante
C++
Estrellas
9.2k
Forks
722
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

### Description

Currently, versions are made upon data ingestion with the following code: https://github.com/activeloopai/deeplake/blob/2ad84c139e91fe6f81da9c8b6d4f48c9d3ee8e73/deeplake/core/vectorstore/deeplake_vectorstore.py#L294

It seems like every time the commit is made, the full dataset of current state is captured as a corresponding version. So, if the user commits a lot, the storage the versions consumes blows up rapidly.

It becomes problematic if the user ingest small data incrementally, i.e., the dataset between versions are almost the same, so consumes space inefficiently.

The canonical solution for this would be to capture only the diff data for each version, but as I'm not acquainted the codebase, don't know whether it is feasible.

So, instead, I suggest to offer an option that users can choose whether they want "auto-commit" or not when ingest a data.

### Use Cases

_No response_

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.