ADORSYS-GIS / ADORSYS-GIS/CoopData

Migrate AI extraction to local/self-hosted models (Ollama) with AWS GPU for production

Aberta
#78 1 comentário 0 reações 1 responsável Reivindicada por @Arielpetit Ver no GitHub
Linguagem predominante
TypeScript
Estrelas
4
Forks
0
Merge médio
20h 22min
PRs com merge (30d)
48

Descrição

# Migrate AI extraction to local/self-hosted models (Ollama) with AWS GPU for production

**Type:** Epic / Story
**Priority:** High
**Labels:** backend, ai, infrastructure, performance
**Story points:** 8

## Description

### Context / Problem

The app currently sends financial-statement files to a cloud LLM API (Google Gemini via `AI_PROVIDER_URL`). Users hit token limits when uploading large files for submission — the full statement text/image exceeds the model's context window and `AI_MAX_TOKENS` output gets truncated (`finish_reason == "max_tokens"`). This blocks real submissions.

Additionally, sending cooperative financial data to a third-party API is a privacy concern (sensitive financial data leaves our infrastructure).

### Goal

Move AI extraction to self-hosted local models served via an OpenAI-compatible endpoint (Ollama), keeping the existing provider-agnostic code path.

Use a single multimodal model for all 4 AI tasks:

1. Financial statement extraction (`map_to_coa`)
2. Image/OCR capture (`vision_capture`)
3. NF Excel header mapping (`map_headers`)
4. Report narrative generation

**Key advantage:** The backend already calls `/chat/completions` via env vars (`AI_PROVIDER_URL`, `AI_MODEL`, `AI_VISION_MODEL`, `AI_MAX_TOKENS`), so this is a config-only change — no code rewrite.

## Proposed Approach

1. **Local testing (dev):** Run Ollama with a small multimodal model (e.g. `qwen2.5-vl:3b` or `minicpm-v:8b`) on the dev machine to validate the pipeline end-to-end.

2. **Production (AWS):** Deploy Ollama + `qwen2.5-vl:32b` (Q4, ~20 GB) on a GPU EC2 instance (`g5.xlarge` / `g6.xlarge`, 24 GB VRAM).

3. **Config switch:** Point `AI_PROVIDER_URL` at the Ollama endpoint in each environment's `.env`. No application code changes expected.

4. **Benchmark:** A/B test `qwen2.5-vl:32b` vs `internvl3:38b` on real statements to pick the best model.

## Acceptance Criteria

- [ ] Local Ollama runs a multimodal model and the extraction pipeline completes end-to-end (correct JSON line items, header mapping, narratives).
- [ ] Production points `AI_PROVIDER_URL` / `AI_MODEL` / `AI_VISION_MODEL` at the AWS GPU Ollama endpoint.
- [ ] A single financial statement extracts successfully with no token-limit errors and no `max_tokens` truncation.
- [ ] Vision/OCR extraction works for image uploads (PNG/JPEG/TIFF).
- [ ] All 4 AI tasks (extraction, OCR, header mapping, narratives) work with the same model.
- [ ] No financial data is sent to third-party APIs in production.
- [ ] Documented in `docs/` (design + runbook) and `.env.example` updated with new AI vars.
- [ ] `cargo clippy` and `cargo test` pass.
- [ ] `npm run lint` / `typecheck` pass.

## Out of Scope

- Rewriting the extraction code (should not be needed).
- Replacing the auth/RBAC system.
- Multi-model orchestration (single model only for now).

## Notes / Risks

- CPU-only dev is slow (~15–45 min/statement on 8B) — acceptable for pipeline validation only.
- GPU cost: `g5.xlarge` ~`$1/hr`; use Spot Instances or auto-shutdown when idle to save cost.
- NVIDIA driver/CUDA must be installed on the GPU instance.
- VRAM: `qwen2.5-vl:32b` Q4 is tight on 24 GB; may need Q3 quantization or a 32 GB instance (`g5.2xlarge`).

Guia de contribuição

Nenhum guia de contribuição indexado para este repositório

Direção de pesquisa

Start by locating the existing AI_PROVIDER_URL, AI_MODEL, AI_VISION_MODEL, and AI_MAX_TOKENS configuration, then run the extraction pipeline against a local Ollama multimodal model. Validate extraction, OCR, header mapping, and narratives end to end; document the design and runbook in docs/, update .env.example, and verify the listed Rust and npm checks.

Escrita pelo modelo de indexação a partir do texto da issue.

Avaliação

Stack de tecnologia
aws, ollama, typescript
Domínio
ai, cloud, infrastructure, performance
Tipo de issue
Funcionalidade
Dificuldade
5/5
Tempo estimado
Mais de uma semana
Status de atividade
Pouca atividade
Clareza
Razoavelmente clara
Facilidade para iniciantes
25/100

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.