mattpocock / mattpocock/evalite

Bug: Evalite Creates Cache Directory Regardless of `cacheEnabled` Setting

Open
#354 2 comments 0 reactions 1 assignee View on GitHub

@mattpocock is already working on this.

Since Nov 28, 2025.

Dominant language
TypeScript
Stars
1.7k
Forks
102
PR merge metrics
No merged PRs in 30d

Description

Description

Evalite unconditionally creates the cache directory at node_modules/.evalite/files/ during initialization, even when cacheEnabled: false is explicitly set in the configuration. This causes failures in read-only filesystem environments like AWS Lambda.

Root Cause

In run-evalite.js, the runEvalite() function creates the files directory before checking the cacheEnabled option:

export const runEvalite = async (opts) => {
    const cwd = opts.cwd ?? process.cwd();
    const filesLocation = path.join(cwd, FILES_LOCATION);  // Line ~135
    await mkdir(filesLocation, { recursive: true });       // Line ~136 - UNCONDITIONAL!
    
    // ... 170+ lines later ...
    
    const cacheEnabled = opts.cacheEnabled ?? config?.cache ?? true;  // Line ~177

The FILES_LOCATION is hardcoded in backend-only-constants.js:

const CACHE_LOCATION = path.join("node_modules", ".evalite");
export const FILES_LOCATION = path.join(CACHE_LOCATION, "files");

Impact

This prevents evalite from running in environments with read-only filesystems, such as:

  • AWS Lambda (where only /tmp is writable)
  • Docker containers with read-only root filesystems
  • CI/CD environments with restricted filesystem permissions

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.