ianarawjo / ianarawjo/ChainForge
Optional EvalPort interop for exported flow response/eval data
- Dominant language
- TypeScript
- Stars
- 3k
- Forks
- 256
- Avg merge
- 4h 40m
- Merged PRs (30d)
- 22
Description
Hi ChainForge team — I maintain [EvalPort](https://github.com/adhabnr-ux/evalport), an open, framework-agnostic JSON spec for portable LLM eval datasets and results (a `TestCase`/`Suite`/`ResultSet` schema with a validator, so a dataset or a graded run can move between tools without hand-writing a converter each time).
I read the actual response/eval types in `chainforge/react-server/src/backend/typing.ts` rather than guessing. Two things line up cleanly with EvalPort's two halves:
- `BaseLLMResponseObject` / `LLMResponse` — `vars` (the template variables filling a prompt) plus `prompt` and `responses` is functionally a graded test case: `vars` is the input, `responses` is the model output.
- `EvaluationResults` (`{ items: EvaluationScore[], dtype: "Numeric" | "Categorical" | "Boolean" | "KeyValue" | ... }`) attached as `eval_res` on an `LLMResponse` is exactly a per-item grader score alongside its output — which is what EvalPort's `ResultSet` entries are for.
So a flow's exported table of `LLMResponse[]` (vars in, responses out, eval_res scored) maps onto an EvalPort `Suite` (the `vars` become each `TestCase`'s input) plus a `ResultSet` (the `responses` + `eval_res` become the graded run) with very little translation — no need to touch the DAG/node execution model itself, just the exported table shape.
Two ways I could see this landing, and I don't have a strong preference:
1. **A standalone `chainforge-openeval-adapter` package** in the EvalPort repo that reads/writes ChainForge's exported `LLMResponse[]` JSON. Zero footprint on this repo.
2. **A small optional export option inside this repo** (e.g. an "Export as EvalPort" alongside the existing export/cache functionality) if you'd rather it live here.
Either way, real tests would validate against EvalPort's actual JSON Schema, not a mock. Filing this as an issue first since I don't see a CONTRIBUTING.md with a different norm — happy to be pointed at one if it exists.
Let me know which direction you'd prefer, or if this isn't a fit for your roadmap right now — no worries either way.
Contributor guide
No contributing guide indexed for this repository
Research direction
Read chainforge/react-server/src/backend/typing.ts and trace the existing exported LLMResponse[] path. Resolve whether the adapter belongs in this repository or EvalPort, then validate the chosen mapping against EvalPort's actual JSON Schema; done means an agreed export/import path with real schema-based tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- ai
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100