Research: how & why Graphify uses an LLM

TL;DR

Graphify is a separate open-source product (not part of this repo) that turns a folder of code/docs/papers/images into a queryable knowledge graph for AI coding assistants — conceptually adjacent to ts_knowledge_graph. The defining difference for us:

Graphify uses an LLM. ts_knowledge_graph does not. Our extraction is pure ts-morph static analysis — there is no LLM SDK in package.json (chalk, commander, kuzu, ts-morph, zod) and no Anthropic/OpenAI/Claude/GPT/embedding call anywhere in src/.

This issue documents, with source-pinned evidence, exactly how and why Graphify uses an LLM, and corrects two things that the project's own README/marketing copy gets wrong (or at least undersells):

1. The LLM's scope is broader than "Markdown / PDF / images / video." YAML, HTML, plain text, .rst, and Office docs (.docx/.xlsx) are also sent to the LLM.
2. "Never raw source code" is almost true but needs two asterisks: raw code bodies/logic are never put in a prompt, but code-derived symbol names do leave the machine via cluster labeling and the optional --dedup-llm tiebreaker.

All line references below are pinned to Graphify commit 1bb30fc (v0.8.38, 2026-06-11).

---

Why we care

We are building a non-LLM static code-graph tool. Graphify is the closest comparable that does use an LLM, so understanding precisely what it delegates to the model and why tells us (a) what static analysis fundamentally cannot do, and (b) where the privacy/cost boundary sits for an LLM-augmented design. The headline finding: Graphify's architecture is the same instinct as ours — deterministic AST extraction as the backbone — with an LLM bolted on only as a second pass for the non-code, semantic material that AST parsing can't reach.

Methodology

git clone --depth 1 https://github.com/safishamsi/graphify → read the source. The package is Python (~35k LOC). The files that matter for LLM usage:

File	LOC	Role
graphify/detect.py	1406	File discovery + type classification (decides code vs doc vs image …)
graphify/extract.py	12059	AST extraction via tree-sitter (the code path)
graphify/llm.py	2142	All LLM backends + the semantic-extraction pass
graphify/__main__.py	4661	CLI orchestration — wires the two paths together
graphify/dedup.py	467	Optional --dedup-llm node-merge tiebreaker
graphify/prs.py	748	prs command — PR triage (calls the LLM directly)
graphify/transcribe.py	184	Video/audio → text (local Whisper)

---

1. The routing rule is a literal extension table

The boundary is not "prose vs. code." It is an extension→bucket lookup in detect.py, and the bucket alone decides which engine runs.

The category constants (detect.py#L28-L33):

# graphify/detect.py:28
CODE_EXTENSIONS = {'.py', '.ts', '.tsx', '.js', '.jsx', '.mjs', '.ejs', '.ets', '.go', '.rs',
    '.java', '.groovy', '.gradle', '.cpp', '.cc', '.cxx', '.c', '.h', '.hpp', '.rb', '.swift',
    '.kt', '.kts', '.cs', '.scala', '.php', '.lua', '.luau', '.toc', '.zig', '.ps1', '.ex',
    '.exs', '.m', '.mm', '.jl', '.vue', '.svelte', '.astro', '.dart', '.v', '.sv', '.svh',
    '.sql', '.r', '.f', ... , '.sh', '.bash', '.json', '.tf', '.tfvars', '.hcl', '.dm', ...,
    '.sln', '.csproj', '.fsproj', '.vbproj', '.razor', '.cshtml', '.cls', '.trigger'}
DOC_EXTENSIONS   = {'.md', '.mdx', '.qmd', '.txt', '.rst', '.html', '.yaml', '.yml'}
PAPER_EXTENSIONS = {'.pdf'}
IMAGE_EXTENSIONS = {'.png', '.jpg', '.jpeg', '.gif', '.webp', '.svg'}
OFFICE_EXTENSIONS = {'.docx', '.xlsx'}
VIDEO_EXTENSIONS  = {'.mp4', '.mov', '.webm', '.mkv', '.avi', '.m4v', '.mp3', '.wav', '.m4a', '.ogg'}

The classifier (detect.py#L384-L411):

# graphify/detect.py:384
def classify_file(path: Path) -> FileType | None:
    if path.name.lower().endswith(".blade.php"):
        return FileType.CODE
    ext = path.suffix.lower()
    if not ext:
        return _shebang_file_type(path)          # extensionless: shebang → CODE
    if ext in CODE_EXTENSIONS:   return FileType.CODE
    if ext in PAPER_EXTENSIONS:  return FileType.PAPER      # (PDF asset-catalog icons → None)
    if ext in IMAGE_EXTENSIONS:  return FileType.IMAGE
    if ext in DOC_EXTENSIONS:
        if _looks_like_paper(path):  return FileType.PAPER  # converted paper
        return FileType.DOCUMENT
    if ext in OFFICE_EXTENSIONS:           return FileType.DOCUMENT
    if ext in GOOGLE_WORKSPACE_EXTENSIONS: return FileType.DOCUMENT
    if ext in VIDEO_EXTENSIONS:            return FileType.VIDEO
    return None

Engine per bucket:

Bucket	Engine	LLM?
CODE	tree-sitter AST (extract.py)	❌ never
DOCUMENT + PAPER + IMAGE	semantic pass (llm.py → extract_corpus_parallel)	✅
VIDEO	local faster-whisper transcription, then transcript → semantic pass	local model; transcript text → ✅

2. The orchestrator wires it together

detect() returns files_by_type, and the CLI splits it into two lists (__main__.py#L4101-L4108):

# graphify/__main__.py:4101
code_files  = [Path(p) for p in files_by_type.get("code", [])]
doc_files   = [Path(p) for p in files_by_type.get("document", [])]
paper_files = [Path(p) for p in files_by_type.get("paper", [])]
image_files = [Path(p) for p in files_by_type.get("image", [])]
...
semantic_files = doc_files + paper_files + image_files      # L4108

code_files → AST extractor (__main__.py#L4209-L4222):

# graphify/__main__.py:4212
if code_files:
    from graphify.extract import extract as _ast_extract
    print(f"[graphify extract] AST extraction on {len(code_files)} code files...")
    ast_result = _ast_extract(code_files, **ast_kwargs)     # no backend, no api_key, no LLM

semantic_files → LLM (__main__.py#L4238-L4283):

# graphify/__main__.py:4252
print(f"[graphify extract] semantic extraction on {len(uncached_paths)} files via {backend}...")
fresh = _extract_corpus_parallel([Path(p) for p in uncached_paths], **corpus_kwargs)

3. The AST/code path has zero LLM calls

grep -niE 'llm|claude|anthropic|openai|prompt|api_key|messages.create' extract.py returns only comments — every hit is prose describing what the other (semantic) pass does. There is no LLM client, no prompt construction, and no fallback to an LLM for code in a language tree-sitter can't parse. An unsupported-language code file simply yields a bare file node. Representative hits:

extract.py:10503   # ... Data JSON is left to the LLM semantic pass.
extract.py:11487   # MCP config files (.mcp.json, claude_desktop_config.json, ...) are routed ...
extract.py:11981   # AST-extracted nodes from semantic/LLM nodes. On a full re-extraction

(Note .json is in CODE_EXTENSIONS, so ordinary JSON goes to tree-sitter; the comment at 10503 concerns data-shaped JSON nodes, not routing whole files to the model.)

4. The semantic pass reads ONLY the files it is handed

The semantic extractor never reaches out to neighboring source. _read_files reads exactly the paths passed in (the semantic_files), wraps each in an <untrusted_source> block, and defangs prompt-injection tokens (llm.py#L449-L467):

# graphify/llm.py:449
def _read_files(paths: list[Path], root: Path) -> str:
    parts: list[str] = []
    for p in paths:
        rel = str(p.relative_to(root))
        content = _file_to_text(p)                      # .pdf → pypdf; else read_text
        parts.append(_wrap_untrusted(rel, content[:_FILE_CHAR_CAP]))
    return "\n\n".join(parts)

My first-pass summary claimed the model receives "code context." That was the web summarizer's paraphrase, not the implementation. _read_files attaches no code.

5. Prompt-injection hardening (tells you what they consider untrusted)

Every semantic file is wrapped and hash-stamped, and known chat-template / jailbreak control tokens are neutralized with a zero-width space (llm.py#L407-L446):

# graphify/llm.py:433
def _wrap_untrusted(rel: str, content: str) -> str:
    sha = hashlib.sha256(content.encode("utf-8", errors="replace")).hexdigest()
    safe = _neutralise_injection_sentinels(content)
    return (f'<untrusted_source path="{rel}" sha256="{sha}">\n'
            f"{safe}\n"
            f"</untrusted_source>")

The sentinel regex defangs </?untrusted_source>, <|im_start|>/<|im_end|>/<|system|>…, <<SYS>>, [INST], and ### system: headers (llm.py#L413-L420). Their system prompt instructs the model to treat everything inside <untrusted_source> as inert data. Implication: doc/PDF/image content is the attack surface they guard — further confirmation that source code is not what flows here.

---

6. Where my earlier "only Markdown / PDF / images / video" was too narrow

The DOCUMENT bucket sent to the LLM is wider than that. It includes:

• .md, .mdx, .qmd
• .txt, .rst (plain text / reStructuredText)
• .html (markup — may contain inline code)
• .yaml, .yml (config-as-text — arguably "source-adjacent")
• .docx, .xlsx (Office)
• Google-Workspace docs

So YAML, HTML, plain text, and Office files are sent to the LLM.

7. Where it is narrower than you'd think

Several things people call "code" or "config" are in CODE_EXTENSIONS, so they go to tree-sitter, not the LLM:

.json, .sql, .sh / .bash, .r, .tf / .tfvars / .hcl, .ps1, .gradle, .csproj / .sln / .vbproj, .trigger, .cls, …

---

8. The actual answer to "does source code reach the LLM?"

Raw code text — function bodies, logic — is never placed in an LLM prompt by any path I found. Files in CODE_EXTENSIONS go to tree-sitter only, on-device, with no fallback (§3), and the semantic extractor never attaches neighboring code (§4).

BUT "nothing derived from your code" is false. Three secondary LLM call sites send code-derived metadata:

Call site	Trigger	What is sent to the LLM	Raw source?
Cluster labeling — generate_community_labels / _community_label_lines (llm.py#L1983)	graph build w/ a backend configured	Node labels only: "Community 3: parseToken, validateToken, …"	❌ names only
--dedup-llm tiebreaker — dedup.py (dedup.py#L434)	opt-in flag	Pairs of labels: "foo" vs "bar"	❌ names only
prs triage — prs.py (prs.py#L598)	prs command	PR metadata: number, status, CI, age, author, title, blast-radius	❌ no diff

Evidence — cluster labeling sends only node labels

# graphify/llm.py:1983  _community_label_lines(...)
for cid, members in ordered[:max_communities]:
    ...
    label = str(G.nodes[nid].get("label", nid)) if nid in G.nodes else str(nid)
    label = label.strip().strip("()")[:_LABEL_MAXLEN]
    ...
    lines.append(f"Community {cid}: {', '.join(names)}")   # e.g. "Community 3: parseToken, validateToken"

Evidence — --dedup-llm sends label pairs only

# graphify/dedup.py:436
pairs_text = "\n".join(f"{i+1}. \"{a['label']}\" vs \"{b['label']}\"" for i, (a, b, _) in enumerate(batch))
prompt = ("For each pair below, answer only 'yes' or 'no': are they the same real-world concept?\n\n"
          f"{pairs_text}\n\n"
          "Reply with one line per pair: '1. yes', '2. no', etc.")
response = _call_llm(prompt, backend=backend, max_tokens=200)

Evidence — prs triage sends PR metadata, not the diff

# graphify/prs.py:590
for pr in candidates:
    lines.append(f"PR #{pr.number} [{pr.status}] CI={pr.ci_status} review={pr.review_decision or 'none'} "
                 f"age={pr.days_old}d author={pr.author}{impact}\n  title: {pr.title}")
prompt = ("You are a senior engineer helping triage a PR review queue. ... \n\n" + "\n\n".join(lines))
# backend == "claude": anthropic.Anthropic(...).messages.stream(model=..., messages=[{"role":"user","content":prompt}])

prs.py does call gh pr diff (--name-only at L222, full diffs at L365) but only to compute blast-radius against the local graph — the diff text is not put into the triage prompt.

Net: your function/class identifiers leave the machine during clustering and optional dedup. That is consistent with Graphify's "never raw source code" wording, but it is not "nothing from your code."

---

9. No bundled model; code-only is fully offline

There is no model shipped with Graphify. It uses whichever backend key is already configured — ANTHROPIC_API_KEY, OPENAI_API_KEY, GEMINI_API_KEY/GOOGLE_API_KEY, MOONSHOT_API_KEY, DEEPSEEK_API_KEY, the claude CLI, AWS Bedrock, Azure, or a local Ollama. A code-only corpus needs no key at all (__main__.py#L4143-L4160):

# graphify/__main__.py:4133
needs_llm = bool(semantic_files) or dedup_llm
...
# "A code-only corpus needs no key."   (error text, L4156)

Backends supported in llm.py: _call_claude (L965), _call_claude_cli (L1030), _call_openai_compat (L831, covers openai/gemini/kimi/deepseek/ollama), _call_azure (L1163), _call_bedrock (L1204).

10. Video/audio is transcribed locally

The VIDEO bucket runs faster-whisper on-device; the audio never leaves the machine. Only the resulting transcript text then flows through the semantic pass (transcribe.py#L1-L24):

# graphify/transcribe.py:1
# Video transcription using faster-whisper
_DEFAULT_MODEL = "base"
def _get_whisper():
    from faster_whisper import WhisperModel
    return WhisperModel

---

11. Corrected one-line characterization

Graphify runs deterministic tree-sitter AST extraction on everything in CODE_EXTENSIONS (incl. .json/.sql/.sh/.r/.tf), entirely on-device with no LLM fallback. It sends to the LLM only the semantic bucket — docs (.md/.mdx/.txt/.rst/.html/.yaml/Office), PDFs, images, and locally-transcribed audio — each wrapped as untrusted, injection-defanged input. Raw code bodies are never sent; code-derived symbol names are, via cluster labeling and the optional --dedup-llm step. No model is bundled; a code-only corpus needs no API key.

12. Why this uses an LLM at all (the rationale)

1. Semantics over prose/visuals. Tree-sitter answers what the code is (functions, imports, call graph) with zero hallucination, but cannot interpret why it was designed that way — that lives in docs, papers, and diagrams, which an AST parser can't read. The LLM extracts concepts/relationships from that unstructured material and links them to code nodes, tagged EXTRACTED / INFERRED / AMBIGUOUS.
2. Query-time token economy. The payoff is that the consuming agent traverses a pre-built graph instead of re-reading raw files (their headline: ~1.7k vs ~123k tokens/query, ≈71×).

13. Implications for ts_knowledge_graph

• Our no-LLM design covers the same backbone Graphify gets from tree-sitter — and we extend it deliberately to .json/config/endpoints via dedicated static extractors, where Graphify either tree-sits them or (for YAML/HTML) hands them to a model.
• If we ever add an optional semantic layer, Graphify's split is a clean reference: keep code on-device; gate any model use behind a configured backend; wrap untrusted doc input; and be explicit that symbol names — not bodies — are what would leave.
• Their public phrasing ("never raw source code", "Markdown/PDF/images") is technically defensible but understates scope (YAML/HTML/Office → LLM) and omits the symbol-name leak. Worth keeping our own docs precise on exactly this axis if we make comparison claims.

---

Appendix — evidence index (all @ 1bb30fc)

Claim	Source
Extension→bucket constants	detect.py#L28-L33
classify_file logic	detect.py#L384-L411
code/semantic split	__main__.py#L4101-L4108
AST path (no LLM)	__main__.py#L4209-L4222
semantic path (LLM)	__main__.py#L4238-L4283
code-only needs no key	__main__.py#L4143-L4160
_read_files (reads only given files)	llm.py#L449-L467
untrusted-source wrap + injection defang	llm.py#L407-L446
cluster labeling (sends labels)	llm.py#L1983-L2007
LLM backends	llm.py#L831-L1247
--dedup-llm (sends label pairs)	dedup.py#L434-L446
prs triage (sends metadata)	prs.py#L590-L622
local Whisper transcription	transcribe.py#L1-L24

Analysis produced by reading Graphify @ 1bb30fcc567a72280f4dc1763947140268f101c5 (v0.8.38). No code in this repo was modified.

Verify the source-pinned findings against Graphify commit 1bb30fc, starting with graphify/detect.py and graphify/main.py for routing, then graphify/llm.py and graphify/dedup.py for LLM inputs. Review graphify/prs.py and graphify/transcribe.py for the remaining call paths. Done means documenting the supported file types, code-versus-semantic boundary, and code-derived metadata exceptions with accurate references.