feat: research ingestion hub — ArXiv → ranked PDF parsing → Claude summarization → Markdown pipeline

## Context

Jack (solo TPM/dev, Mac Studio M4) needs a daily pipeline that queries ArXiv across cs.AI, cs.LG, and cs.MA, scores and filters to the top 25 papers by relevance, extracts PDF text via PyMuPDF, and produces \"I AM JACK\" format Markdown summaries via Claude 3.5 Sonnet. Summaries are stored in \`~/NovaOS/data/research/YYYY-MM-DD/\` and served by a FastAPI endpoint for the NovaOS dashboard. Today: no pipeline exists. Papers are read manually.

---

## Current State

\`~/NovaOS/\` does not exist on disk. \`python3.11\` (3.11.15) is available alongside the system 3.9.6. \`fastapi\`, \`pydantic\`, and \`anthropic\` are installed; \`arxiv\` and \`pymupdf\` are not. \`uv\` is available for dependency management. Verified 2026-05-27.

---

## Proposed Change

A four-stage Python pipeline exposed as a gstack skill (\`/research\`) and a \`launchd\` daily job at 08:00:

```
ArXiv API → Rank (Top 25) → PDF download → PyMuPDF extract → Claude summarize → SQLite + Markdown → FastAPI
```

### Project Layout

```
~/NovaOS/
├── bin/
│ └── research-engine # Python 3.11 CLI entry point
├── src/
│ ├── research_engine.py # orchestrator
│ ├── ingestor.py # ArXiv API client
│ ├── ranker.py # scoring + Top 25 cut
│ ├── pdf_parser.py # PyMuPDF extraction
│ ├── summarizer.py # Claude 3.5 Sonnet wrapper
│ └── db.py # SQLite layer
├── api/
│ └── server.py # FastAPI /papers endpoint
├── config/
│ ├── watchlist.json # editable: authors, labs, keywords
│ └── prompts.yaml # editable: LLM prompt templates
├── data/
│ └── research/
│ ├── research_hub.db # SQLite dedup + audit index
│ └── YYYY-MM-DD/
│ └── .md # one file per summarized paper
├── tests/
└── pyproject.toml # requires-python = ">=3.11", managed by uv

~/.claude/skills/gstack/
└── research/
└── SKILL.md # /research gstack skill
```

---

### SQLite Schema

```sql
CREATE TABLE papers (
arxiv_id TEXT PRIMARY KEY,
title TEXT NOT NULL,
authors TEXT NOT NULL, -- JSON array
categories TEXT NOT NULL, -- JSON array
abstract TEXT NOT NULL,
pdf_url TEXT NOT NULL,
published_at TEXT NOT NULL, -- ISO-8601
fetched_at TEXT NOT NULL, -- ISO-8601
rank_score REAL NOT NULL DEFAULT 0.0,
processed_at TEXT, -- NULL until summarized
summary_path TEXT, -- absolute path to .md file
ingested INTEGER NOT NULL DEFAULT 0,
summarized INTEGER NOT NULL DEFAULT 0,
synced_to_notion INTEGER NOT NULL DEFAULT 0
);

CREATE TABLE audit_logs (
id INTEGER PRIMARY KEY AUTOINCREMENT,
arxiv_id TEXT,
event_type TEXT, -- 'rank' | 'parse' | 'summarize'
model_ver TEXT,
tokens_in INTEGER,
tokens_out INTEGER,
latency_ms REAL,
status TEXT, -- 'success' | 'fail'
error_msg TEXT,
created_at TEXT DEFAULT CURRENT_TIMESTAMP
);

CREATE INDEX idx_published_at ON papers(published_at DESC);
CREATE INDEX idx_rank_score ON papers(rank_score DESC);
```

---

### Ranking Algorithm

```python
def score_paper(paper: ArxivPaper, watchlist: Watchlist) -> float:
score = 0.0
# Watchlist match: capped at one bonus per paper
authors_lower = [a.lower() for a in paper.authors]
watch_terms = [w.lower() for w in watchlist.authors + watchlist.labs]
if any(w in a for a in authors_lower for w in watch_terms):
score += 50.0
# Keyword density in title + abstract
text = (paper.title + " " + paper.abstract).lower()
for kw in [k.lower() for k in watchlist.keywords]:
score += text.count(kw) * 10.0
# Recency decay: exponential, half-life = 2 days
days_old = (datetime.utcnow() - paper.published_at).total_seconds() / 86400
score *= math.exp(-days_old * math.log(2) / 2.0)
return score
```

---

### Config Schemas

**\`watchlist.json\`**
```json
{
"authors": ["andrej karpathy"],
"labs": ["openclaw", "hermes", "autogen", "deepmind", "anthropic"],
"keywords": ["autonomous agents", "mlops automation", "enterprise ai",
"agentic workflows", "agentic", "mlops"]
}
```
Keys are stored lower-cased; comparison is always case-insensitive.

**\`prompts.yaml\`**
```yaml
summarize:
model: claude-3-5-sonnet-20241022
max_tokens: 1500
temperature: 0.1
system: |
You are a technical research synthesizer for an AI TPM.
Use precise technical nomenclature. Prioritize scannability.
Never pad. Every sentence must carry information.
template: |
Synthesize the following paper into a high-density Markdown summary.

## {title}
**ArXiv:** {arxiv_id} | **Category:** {categories} | **PDF:** {pdf_url}

### TL;DR
[2 sentences. Core innovation + primary result.]

### Quantified Impact
[Bullets only. Format: "Dataset: X% over baseline Y". No qualitative claims.]

### Architectural Overview
[3-5 bullets. Name models, loss functions, orchestration frameworks explicitly.]

### Risk & Lifecycle Assessment
[2-4 bullets. Limitations, failure modes, security flags, reproducibility.]

---
PAPER TEXT:
{paper_text}
```

---

### PDF Text Extraction

```python
def extract_pdf_text(pdf_path: Path, max_chars: int = 50_000) -> str:
with fitz.open(pdf_path) as doc:
# sort=True preserves logical reading order across multi-column layouts
return "".join(page.get_text("text", sort=True) for page in doc)[:max_chars]
```

\`max_chars=50_000\` keeps each Claude call under ~13K input tokens. Estimated cost: ~$0.01/paper × 25 papers/day = **~$0.25/day (~$7.50/month)**.

---

### CLI Interface

```bash
research-engine run # daily scheduled run
research-engine run --query "multi-agent RL" # ad-hoc keyword override
research-engine run --from 2026-05-20 --to 2026-05-27 # backfill
research-engine status # show today's run state
research-engine serve --port 8765 # start FastAPI server
```

---

### FastAPI \`/papers\` Endpoint

```
GET /papers
?date=2026-05-27 (default: today)
&category=cs.AI (optional)
&min_score=20.0 (optional)
&summarized=true (default: true)

Response: [{
arxiv_id: string,
title: string,
categories: string[],
rank_score: number,
summary_path: string,
summary_markdown: string,
pdf_url: string,
published_at: string
}]

GET /papers/:arxiv_id -- single paper
GET /health -- last run timestamp + paper count
```

---

### launchd Plists (\`~/Library/LaunchAgents/\`)

| Plist | Trigger | ProgramArguments |
|---|---|---|
| \`com.novaos.research-daily.plist\` | 08:00 daily | \`uv run --project /Users/jack/NovaOS /Users/jack/NovaOS/bin/research-engine run\` |
| \`com.novaos.research-api.plist\` | login | \`uv run --project /Users/jack/NovaOS /Users/jack/NovaOS/bin/research-engine serve --port 8765\` |

Absolute paths required — \`launchd\` does not inherit shell \`PATH\`.

---

## Acceptance Criteria

1. \`research-engine run\` completes in under 15 minutes for 25 papers on M4
2. Each output \`.md\` contains all four section headers (\`### TL;DR\`, \`### Quantified Impact\`, \`### Architectural Overview\`, \`### Risk & Lifecycle Assessment\`) — verified by regex
3. SQLite skips any \`arxiv_id\` where \`processed_at IS NOT NULL\` — no duplicate summaries on re-run
4. \`GET /papers\` returns correctly structured JSON for all \`summarized=1\` rows
5. \`launchd\` daily job fires at 08:00 and writes to \`~/NovaOS/data/research/YYYY-MM-DD/\`
6. A watchlist-matched paper scores higher than a same-date non-matched paper (unit test)
7. \`--query\` flag narrows ArXiv fetch to keyword-matching results only
8. Editing \`prompts.yaml\` changes LLM output on next run without touching Python code
9. Every \`summarize\` event writes a row to \`audit_logs\` with non-null \`tokens_in\` and \`tokens_out\`
10. A second \`research-engine run\` within the same 24-hour window makes 0 additional Claude API calls when the Top 25 set is unchanged
11. \`GET /papers\` returns its JSON payload in under 200ms on a 100-row table

---

## Testing Plan

| Layer | What | +Tests |
|---|---|---|
| Unit | \`score_paper()\` — watchlist match, keyword density, recency decay, zero case | +4 |
| Unit | SQLite dedup: skip if \`processed_at\` is set | +2 |
| Unit | PyMuPDF extraction with fixture PDF (multi-column layout) | +2 |
| Unit | \`prompts.yaml\` template rendering with all fields | +2 |
| Integration | ArXiv API → rank → Top 25 selection (live, 1 category) | +1 |
| Integration | Full mock run: 3 papers → 3 \`.md\` files + 3 SQLite rows + 3 audit_log rows | +1 |
| E2E | \`research-engine run\` on live ArXiv, 1 paper, verify file written and 4 headers present | +1 |

---

## Rollback Plan

\`research-engine run\` is additive-only — writes files and SQLite rows, never deletes. To rollback: delete \`~/NovaOS/data/research/YYYY-MM-DD/\` and run:

```sql
DELETE FROM papers WHERE fetched_at > '';
DELETE FROM audit_logs WHERE created_at > '';
```

Stop the API server: \`launchctl unload ~/Library/LaunchAgents/com.novaos.research-api.plist\`

---

## Effort Estimate

| Component | Human | CC+gstack |
|---|---|---|
| Core pipeline (\`ingestor\`, \`ranker\`, \`parser\`, \`summarizer\`) | 3 days | 20 min |
| SQLite layer + audit_logs | 4 hours | 5 min |
| FastAPI server | 4 hours | 5 min |
| gstack skill (\`research/SKILL.md\`) | 2 hours | 5 min |
| launchd plists | 1 hour | 2 min |
| Tests | 1 day | 10 min |
| Config files + bootstrap | 1 hour | 3 min |
| **Total** | **~5 days** | **~50 min** |

---

## Files Reference

| File | Change |
|---|---|
| \`~/NovaOS/bin/research-engine\` | New Python 3.11 CLI |
| \`~/NovaOS/src/research_engine.py\` | Orchestrator |
| \`~/NovaOS/src/ingestor.py\` | ArXiv API client |
| \`~/NovaOS/src/ranker.py\` | Scoring + Top 25 |
| \`~/NovaOS/src/pdf_parser.py\` | PyMuPDF extraction |
| \`~/NovaOS/src/summarizer.py\` | Claude wrapper |
| \`~/NovaOS/src/db.py\` | SQLite layer |
| \`~/NovaOS/api/server.py\` | FastAPI \`/papers\` |
| \`~/NovaOS/config/watchlist.json\` | Author/lab/keyword config |
| \`~/NovaOS/config/prompts.yaml\` | LLM prompt config |
| \`~/NovaOS/pyproject.toml\` | Python 3.11+, \`uv\`-managed |
| \`~/Library/LaunchAgents/com.novaos.research-daily.plist\` | 08:00 daily job |
| \`~/Library/LaunchAgents/com.novaos.research-api.plist\` | API server daemon |
| \`~/.claude/skills/gstack/research/SKILL.md\` | gstack skill |
| \`~/NovaOS/tests/\` | Test suite |

---

## Out of Scope

- Ollama local fallback (Phase 2 — \`synced_to_notion\` flag pre-wired in schema)
- Notion sync (Phase 2)
- Multi-user auth / RBAC
- Slack/Jira integration
- Push notifications
- Marker PDF parser (Phase 2 upgrade path from PyMuPDF)
- NovaOS React dashboard (consumer, not built here)

---

*Spec authored via \`/spec\` — gstack v1.51.0.0*