0xPlaygrounds / 0xPlaygrounds/rig

feat: CSV loader support

Offen
#29 1 Kommentar 2 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Rust
Sterne
8.6k
Forks
959
Ø Merge
4 Std. 32 Min.
Gemergte PRs (30 T.)
117

Beschreibung

- [x] I have looked for existing issues (including closed) about this

## Feature Request: Implement CSV Loader for Document Processing
`rig-core/src/loaders/csv.rs` -> `CsvFileLoader`

### Motivation

As users of Rig often need to work with structured data stored in CSV files, we need a way to easily load and process CSV documents for use in RAG systems and other document processing tasks. A CSV loader would allow users to incorporate tabular data into their NLP pipelines, enhancing the versatility of Rig for various use cases such as data analysis, information retrieval, and content summarization.

### Proposal

Implement a `CsvLoader` struct that implements the `DocumentLoader` trait. The loader should:

1. Accept a file path to a CSV file.
2. Parse the CSV file using the `csv` crate.
3. Convert the CSV data into a format suitable for embedding and further processing within Rig.
4. Handle potential errors such as file not found, parsing errors, or invalid CSV structures.
5. Provide options for customization, such as specifying delimiters or handling headers.

The implementation should focus on converting CSV data into a single document for embedding, with each row formatted as "header: value" pairs, separated by newlines.

### Alternatives

1. **Row-based Embedding**: Instead of creating a single document, we could create separate embeddings for each row. This would allow for more granular retrieval but might increase processing time and storage requirements.

Drawbacks: Increased complexity in implementation and potential performance impact for large CSV files.

2. **Using pandas-like library**: We could use a more robust data processing library like polars to handle CSV files, which might offer more advanced features for data manipulation.

Drawbacks: Introduces a heavier dependency, which might not be necessary for simple CSV processing.

3. **Custom parsing without csv crate**: We could implement CSV parsing without relying on the `csv` crate, giving us more control over the parsing process.

Drawbacks: Reinventing the wheel, potentially introducing bugs, and increasing maintenance burden.

The proposed solution was chosen because it offers a good balance between simplicity, performance, and flexibility. It leverages the well-tested `csv` crate for parsing while allowing for future enhancements if more advanced features are needed.

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

The issue specifies implementing a CsvFileLoader in rig-core/src/loaders/csv.rs. Start by examining the existing DocumentLoader trait and other loaders in the same directory to understand the interface. Use the csv crate for parsing. The goal is to read a CSV file and format each row as 'header: value' pairs into a single document. Check for existing tests for loaders to understand how to test the new implementation.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
rust
Bereich
backend
Issue-Typ
Feature
Schwierigkeit
3/5
Geschätzter Aufwand
1-2 Tage
Aktivitätsstatus
Veraltet
Klarheit
Klar beschrieben
Anfängerfreundlichkeit
65/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.