ADORSYS-GIS / ADORSYS-GIS/CoopData
Migrate AI extraction to local/self-hosted models (Ollama) with AWS GPU for production
- Ngôn ngữ chính
- TypeScript
- Star
- 4
- Fork
- 0
- Merge trung bình
- 21 giờ 50 phút
- Pull request đã merge (30 ngày)
- 45
Mô tả
# Migrate AI extraction to local/self-hosted models (Ollama) with AWS GPU for production
**Type:** Epic / Story
**Priority:** High
**Labels:** backend, ai, infrastructure, performance
**Story points:** 8
## Description
### Context / Problem
The app currently sends financial-statement files to a cloud LLM API (Google Gemini via `AI_PROVIDER_URL`). Users hit token limits when uploading large files for submission — the full statement text/image exceeds the model's context window and `AI_MAX_TOKENS` output gets truncated (`finish_reason == "max_tokens"`). This blocks real submissions.
Additionally, sending cooperative financial data to a third-party API is a privacy concern (sensitive financial data leaves our infrastructure).
### Goal
Move AI extraction to self-hosted local models served via an OpenAI-compatible endpoint (Ollama), keeping the existing provider-agnostic code path.
Use a single multimodal model for all 4 AI tasks:
1. Financial statement extraction (`map_to_coa`)
2. Image/OCR capture (`vision_capture`)
3. NF Excel header mapping (`map_headers`)
4. Report narrative generation
**Key advantage:** The backend already calls `/chat/completions` via env vars (`AI_PROVIDER_URL`, `AI_MODEL`, `AI_VISION_MODEL`, `AI_MAX_TOKENS`), so this is a config-only change — no code rewrite.
## Proposed Approach
1. **Local testing (dev):** Run Ollama with a small multimodal model (e.g. `qwen2.5-vl:3b` or `minicpm-v:8b`) on the dev machine to validate the pipeline end-to-end.
2. **Production (AWS):** Deploy Ollama + `qwen2.5-vl:32b` (Q4, ~20 GB) on a GPU EC2 instance (`g5.xlarge` / `g6.xlarge`, 24 GB VRAM).
3. **Config switch:** Point `AI_PROVIDER_URL` at the Ollama endpoint in each environment's `.env`. No application code changes expected.
4. **Benchmark:** A/B test `qwen2.5-vl:32b` vs `internvl3:38b` on real statements to pick the best model.
## Acceptance Criteria
- [ ] Local Ollama runs a multimodal model and the extraction pipeline completes end-to-end (correct JSON line items, header mapping, narratives).
- [ ] Production points `AI_PROVIDER_URL` / `AI_MODEL` / `AI_VISION_MODEL` at the AWS GPU Ollama endpoint.
- [ ] A single financial statement extracts successfully with no token-limit errors and no `max_tokens` truncation.
- [ ] Vision/OCR extraction works for image uploads (PNG/JPEG/TIFF).
- [ ] All 4 AI tasks (extraction, OCR, header mapping, narratives) work with the same model.
- [ ] No financial data is sent to third-party APIs in production.
- [ ] Documented in `docs/` (design + runbook) and `.env.example` updated with new AI vars.
- [ ] `cargo clippy` and `cargo test` pass.
- [ ] `npm run lint` / `typecheck` pass.
## Out of Scope
- Rewriting the extraction code (should not be needed).
- Replacing the auth/RBAC system.
- Multi-model orchestration (single model only for now).
## Notes / Risks
- CPU-only dev is slow (~15–45 min/statement on 8B) — acceptable for pipeline validation only.
- GPU cost: `g5.xlarge` ~`$1/hr`; use Spot Instances or auto-shutdown when idle to save cost.
- NVIDIA driver/CUDA must be installed on the GPU instance.
- VRAM: `qwen2.5-vl:32b` Q4 is tight on 24 GB; may need Q3 quantization or a 32 GB instance (`g5.2xlarge`).
Hướng dẫn đóng góp
Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này
Đánh giá
Issue này chưa được đánh giá.