0xPlaygrounds / 0xPlaygrounds/rig

feat: CSV loader support

Aperta
#29 1 commento 2 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Rust
Stelle
8.6k
Fork
959
Merge medio
4h 32m
PR unite (30g)
117

Descrizione

- [x] I have looked for existing issues (including closed) about this

## Feature Request: Implement CSV Loader for Document Processing
`rig-core/src/loaders/csv.rs` -> `CsvFileLoader`

### Motivation

As users of Rig often need to work with structured data stored in CSV files, we need a way to easily load and process CSV documents for use in RAG systems and other document processing tasks. A CSV loader would allow users to incorporate tabular data into their NLP pipelines, enhancing the versatility of Rig for various use cases such as data analysis, information retrieval, and content summarization.

### Proposal

Implement a `CsvLoader` struct that implements the `DocumentLoader` trait. The loader should:

1. Accept a file path to a CSV file.
2. Parse the CSV file using the `csv` crate.
3. Convert the CSV data into a format suitable for embedding and further processing within Rig.
4. Handle potential errors such as file not found, parsing errors, or invalid CSV structures.
5. Provide options for customization, such as specifying delimiters or handling headers.

The implementation should focus on converting CSV data into a single document for embedding, with each row formatted as "header: value" pairs, separated by newlines.

### Alternatives

1. **Row-based Embedding**: Instead of creating a single document, we could create separate embeddings for each row. This would allow for more granular retrieval but might increase processing time and storage requirements.

Drawbacks: Increased complexity in implementation and potential performance impact for large CSV files.

2. **Using pandas-like library**: We could use a more robust data processing library like polars to handle CSV files, which might offer more advanced features for data manipulation.

Drawbacks: Introduces a heavier dependency, which might not be necessary for simple CSV processing.

3. **Custom parsing without csv crate**: We could implement CSV parsing without relying on the `csv` crate, giving us more control over the parsing process.

Drawbacks: Reinventing the wheel, potentially introducing bugs, and increasing maintenance burden.

The proposed solution was chosen because it offers a good balance between simplicity, performance, and flexibility. It leverages the well-tested `csv` crate for parsing while allowing for future enhancements if more advanced features are needed.

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

The issue specifies implementing a CsvFileLoader in rig-core/src/loaders/csv.rs. Start by examining the existing DocumentLoader trait and other loaders in the same directory to understand the interface. Use the csv crate for parsing. The goal is to read a CSV file and format each row as 'header: value' pairs into a single document. Check for existing tests for loaders to understand how to test the new implementation.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
rust
Ambito
backend
Tipo di issue
Funzionalità
Difficoltà
3/5
Tempo stimato
1-2 giorni
Stato di attività
Ferma
Chiarezza
Specificata chiaramente
Idoneità per principianti
65/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.