ianarawjo / ianarawjo/ChainForge

Optional EvalPort interop for exported flow response/eval data

Open
#414 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
3k
Forks
256
Avg merge
4h 40m
Merged PRs (30d)
22

Description

Hi ChainForge team — I maintain [EvalPort](https://github.com/adhabnr-ux/evalport), an open, framework-agnostic JSON spec for portable LLM eval datasets and results (a `TestCase`/`Suite`/`ResultSet` schema with a validator, so a dataset or a graded run can move between tools without hand-writing a converter each time).

I read the actual response/eval types in `chainforge/react-server/src/backend/typing.ts` rather than guessing. Two things line up cleanly with EvalPort's two halves:

- `BaseLLMResponseObject` / `LLMResponse` — `vars` (the template variables filling a prompt) plus `prompt` and `responses` is functionally a graded test case: `vars` is the input, `responses` is the model output.
- `EvaluationResults` (`{ items: EvaluationScore[], dtype: "Numeric" | "Categorical" | "Boolean" | "KeyValue" | ... }`) attached as `eval_res` on an `LLMResponse` is exactly a per-item grader score alongside its output — which is what EvalPort's `ResultSet` entries are for.

So a flow's exported table of `LLMResponse[]` (vars in, responses out, eval_res scored) maps onto an EvalPort `Suite` (the `vars` become each `TestCase`'s input) plus a `ResultSet` (the `responses` + `eval_res` become the graded run) with very little translation — no need to touch the DAG/node execution model itself, just the exported table shape.

Two ways I could see this landing, and I don't have a strong preference:

1. **A standalone `chainforge-openeval-adapter` package** in the EvalPort repo that reads/writes ChainForge's exported `LLMResponse[]` JSON. Zero footprint on this repo.
2. **A small optional export option inside this repo** (e.g. an "Export as EvalPort" alongside the existing export/cache functionality) if you'd rather it live here.

Either way, real tests would validate against EvalPort's actual JSON Schema, not a mock. Filing this as an issue first since I don't see a CONTRIBUTING.md with a different norm — happy to be pointed at one if it exists.

Let me know which direction you'd prefer, or if this isn't a fit for your roadmap right now — no worries either way.

Contributor guide

No contributing guide indexed for this repository

Research direction

Read chainforge/react-server/src/backend/typing.ts and trace the existing exported LLMResponse[] path. Resolve whether the adapter belongs in this repository or EvalPort, then validate the chosen mapping against EvalPort's actual JSON Schema; done means an agreed export/import path with real schema-based tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
ai
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.