0xPlaygrounds / 0xPlaygrounds/rig

feat: CSV loader support

Abierto
#29 1 comentario 2 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Rust
Estrellas
8.6k
Forks
959
Merge medio
4 h 32 min
PR fusionados (30 d)
117

Descripción

- [x] I have looked for existing issues (including closed) about this

## Feature Request: Implement CSV Loader for Document Processing
`rig-core/src/loaders/csv.rs` -> `CsvFileLoader`

### Motivation

As users of Rig often need to work with structured data stored in CSV files, we need a way to easily load and process CSV documents for use in RAG systems and other document processing tasks. A CSV loader would allow users to incorporate tabular data into their NLP pipelines, enhancing the versatility of Rig for various use cases such as data analysis, information retrieval, and content summarization.

### Proposal

Implement a `CsvLoader` struct that implements the `DocumentLoader` trait. The loader should:

1. Accept a file path to a CSV file.
2. Parse the CSV file using the `csv` crate.
3. Convert the CSV data into a format suitable for embedding and further processing within Rig.
4. Handle potential errors such as file not found, parsing errors, or invalid CSV structures.
5. Provide options for customization, such as specifying delimiters or handling headers.

The implementation should focus on converting CSV data into a single document for embedding, with each row formatted as "header: value" pairs, separated by newlines.

### Alternatives

1. **Row-based Embedding**: Instead of creating a single document, we could create separate embeddings for each row. This would allow for more granular retrieval but might increase processing time and storage requirements.

Drawbacks: Increased complexity in implementation and potential performance impact for large CSV files.

2. **Using pandas-like library**: We could use a more robust data processing library like polars to handle CSV files, which might offer more advanced features for data manipulation.

Drawbacks: Introduces a heavier dependency, which might not be necessary for simple CSV processing.

3. **Custom parsing without csv crate**: We could implement CSV parsing without relying on the `csv` crate, giving us more control over the parsing process.

Drawbacks: Reinventing the wheel, potentially introducing bugs, and increasing maintenance burden.

The proposed solution was chosen because it offers a good balance between simplicity, performance, and flexibility. It leverages the well-tested `csv` crate for parsing while allowing for future enhancements if more advanced features are needed.

Guía de contribución

Abrir la guía de contribución

Línea de trabajo

The issue specifies implementing a CsvFileLoader in rig-core/src/loaders/csv.rs. Start by examining the existing DocumentLoader trait and other loaders in the same directory to understand the interface. Use the csv crate for parsing. The goal is to read a CSV file and format each row as 'header: value' pairs into a single document. Check for existing tests for loaders to understand how to test the new implementation.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
rust
Área
backend
Tipo de issue
Nueva funcionalidad
Dificultad
3/5
Tiempo estimado
1-2 días
Estado de actividad
Estancado
Claridad
Bien especificado
Aptitud para principiantes
65/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.