mattpocock / mattpocock/evalite
possible regression: cannot use `@nuxt/test-utils` `defineVitestProject` directly in evalite.config.ts
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 1.7k
- Forks
- 102
- PR merge metrics
- No merged PRs in 30d
Description
Since recent Evalite versions, using defineVitestProject() from @nuxt/test-utils directly in evalite.config.ts is no longer practical in my setup, although I haven’t yet identified the exact change that caused this (could be on Evalite, Vitest, or Nuxt test-utils side)
Before, I could plug Nuxt test-utils config directly.
Now I need to manually strip/hoist fields (setupFiles, testTimeout, maxConcurrency, include) to avoid Evalite validation/override behavior.
Environment
- evalite:
1.0.0-beta.16 - vitest:
4.1.x - @nuxt/test-utils:
4.0.x - nuxt:
4.4.x - OS: Windows
Reproduction
evalite.config.ts:
import { defineConfig } from 'evalite/config'
import { defineVitestProject } from '@nuxt/test-utils/config'
const ai = await defineVitestProject({
test: {
name: 'ai',
include: ['test/ai/*.{test,eval}.ts'],
environment: 'nuxt'
}
})
export default defineConfig({
viteConfig: {
test: {
projects: [ai]
}
}
})
test.eval.ts:
import { evalite } from "evalite";
evalite("My Eval", {
data: [{ input: "Hello" }],
task: async (input) => {
return input + " Paris!";
},
scorers: [
{
name: "Contains Paris",
description: "Checks if the output contains the word 'Paris'.",
scorer: ({ output }) => {
return output.includes("Paris") ? 1 : 0;
},
},
],
});
Run:
npx evalite run test.eval.ts
Actual behavior
test.projectspath is not usable in practice with Evalite’s inline Vitest creation.- Some fields (e.g. setupFiles, testTimeout, maxConcurrency, include) appear to be either restricted or overridden by Evalite, which makes direct reuse of existing Vitest project config more difficult.
- Evalite also overrides
include/config/watch/mode, which makes direct reuse of project config difficult.
I might be missing something in how these pieces are expected to integrate, so happy to be corrected if there is a supported pattern I overlooked.
Expected behavior
One of:
- Existing Vitest project/workspace configs should compose directly with Evalite
- Evalite should provide an official adapter/helper for Vitest project configs
- The supported integration pattern should be clearly documented, including how to migrate existing Vitest-based setups
Current workaround
I must split and sanitize config manually:
- hoist
setupFilesto Evalite root - strip Evalite-forbidden/overridden
testfields - pass only top-level Vite/Nuxt parts into
viteConfig
import { defineConfig } from 'evalite/config'
import { defineVitestProject } from '@nuxt/test-utils/config'
const ai = await defineVitestProject({
test: {
name: 'ai',
include: ['test/ai/*.{test,eval}.ts'],
environment: 'nuxt',
hookTimeout: 60_000
}
})
// Evalite-safe: strips workspace projects, hoists Nuxt plugins top-level
const { test: aiTestRaw, ...aiVite } = ai
const { setupFiles, testTimeout: _tt, maxConcurrency: _mc, include: _inc, ...aiTest } = aiTestRaw ?? {}
const evaliteSetupFiles = Array.isArray(setupFiles) ? setupFiles : setupFiles ? [setupFiles] : []
const evaliteViteConfig = { ...aiVite, test: aiTest }
export default defineConfig({
scoreThreshold: 75,
testTimeout: 10 * 60 * 1000,
setupFiles: evaliteSetupFiles,
viteConfig: evaliteViteConfig
})
This workaround works for now, but feels brittle and requires manual reshaping of an already valid Vitest config.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the issue with evalite.config.ts, test.eval.ts, and npx evalite run test.eval.ts. Trace Evalite's inline Vitest creation and its handling of the defineVitestProject result, then compare that with the listed manual reshaping workaround. Done means direct project composition works or the supported integration pattern is documented and covered by a regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- nuxt, typescript
- Domain
- testing-qa, tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100