ADORSYS-GIS / ADORSYS-GIS/CoopData

Migrate AI extraction to local/self-hosted models (Ollama) with AWS GPU for production

Đang mở
#78 1 bình luận 0 reaction 1 người được giao Được @Arielpetit nhận Xem trên GitHub
Ngôn ngữ chính
TypeScript
Star
4
Fork
0
Merge trung bình
21 giờ 50 phút
Pull request đã merge (30 ngày)
45

Mô tả

# Migrate AI extraction to local/self-hosted models (Ollama) with AWS GPU for production

**Type:** Epic / Story
**Priority:** High
**Labels:** backend, ai, infrastructure, performance
**Story points:** 8

## Description

### Context / Problem

The app currently sends financial-statement files to a cloud LLM API (Google Gemini via `AI_PROVIDER_URL`). Users hit token limits when uploading large files for submission — the full statement text/image exceeds the model's context window and `AI_MAX_TOKENS` output gets truncated (`finish_reason == "max_tokens"`). This blocks real submissions.

Additionally, sending cooperative financial data to a third-party API is a privacy concern (sensitive financial data leaves our infrastructure).

### Goal

Move AI extraction to self-hosted local models served via an OpenAI-compatible endpoint (Ollama), keeping the existing provider-agnostic code path.

Use a single multimodal model for all 4 AI tasks:

1. Financial statement extraction (`map_to_coa`)
2. Image/OCR capture (`vision_capture`)
3. NF Excel header mapping (`map_headers`)
4. Report narrative generation

**Key advantage:** The backend already calls `/chat/completions` via env vars (`AI_PROVIDER_URL`, `AI_MODEL`, `AI_VISION_MODEL`, `AI_MAX_TOKENS`), so this is a config-only change — no code rewrite.

## Proposed Approach

1. **Local testing (dev):** Run Ollama with a small multimodal model (e.g. `qwen2.5-vl:3b` or `minicpm-v:8b`) on the dev machine to validate the pipeline end-to-end.

2. **Production (AWS):** Deploy Ollama + `qwen2.5-vl:32b` (Q4, ~20 GB) on a GPU EC2 instance (`g5.xlarge` / `g6.xlarge`, 24 GB VRAM).

3. **Config switch:** Point `AI_PROVIDER_URL` at the Ollama endpoint in each environment's `.env`. No application code changes expected.

4. **Benchmark:** A/B test `qwen2.5-vl:32b` vs `internvl3:38b` on real statements to pick the best model.

## Acceptance Criteria

- [ ] Local Ollama runs a multimodal model and the extraction pipeline completes end-to-end (correct JSON line items, header mapping, narratives).
- [ ] Production points `AI_PROVIDER_URL` / `AI_MODEL` / `AI_VISION_MODEL` at the AWS GPU Ollama endpoint.
- [ ] A single financial statement extracts successfully with no token-limit errors and no `max_tokens` truncation.
- [ ] Vision/OCR extraction works for image uploads (PNG/JPEG/TIFF).
- [ ] All 4 AI tasks (extraction, OCR, header mapping, narratives) work with the same model.
- [ ] No financial data is sent to third-party APIs in production.
- [ ] Documented in `docs/` (design + runbook) and `.env.example` updated with new AI vars.
- [ ] `cargo clippy` and `cargo test` pass.
- [ ] `npm run lint` / `typecheck` pass.

## Out of Scope

- Rewriting the extraction code (should not be needed).
- Replacing the auth/RBAC system.
- Multi-model orchestration (single model only for now).

## Notes / Risks

- CPU-only dev is slow (~15–45 min/statement on 8B) — acceptable for pipeline validation only.
- GPU cost: `g5.xlarge` ~`$1/hr`; use Spot Instances or auto-shutdown when idle to save cost.
- NVIDIA driver/CUDA must be installed on the GPU instance.
- VRAM: `qwen2.5-vl:32b` Q4 is tight on 24 GB; may need Q3 quantization or a 32 GB instance (`g5.2xlarge`).

Hướng dẫn đóng góp

Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.