github / github/gh-aw

Declarative memory schema constraints

Open
#58,144 1 comment 0 reactions 1 assignee Claimed by @lpcox View on GitHub
enhancement workflows
Dominant language
Go
Stars
5.1k
Forks
541
Avg merge
5h 46m
Merged PRs (30d)
760

Description

## Problem

`repo-memory`, `cache-memory`, and `drive-memory` support `validation.script`, but even ordinary JSON shape validation must currently be implemented as imperative JavaScript. This duplicates a schema validator in every workflow, produces inconsistent diagnostics, and makes constraints harder to review and maintain.

For example, [`github/github-automation`'s `ci-perf` workflow](https://github.com/github/github-automation/blob/main/.github/workflows/ci-perf.md?plain=1#L138-L154) currently contains:

```yaml
validation:
timeout-minutes: 1
script: |
const fail = message => { throw new Error(`notes.json: ${message}`); };
const data = JSON.parse(fs.readFileSync(path.join(memoryRoot, "notes.json"), "utf8"));
const isObject = value => value !== null && typeof value === "object" && !Array.isArray(value);
if (!isObject(data)) fail("must contain an object");
if (Object.keys(data).sort().join(",") !== "investigations,notes") fail("must contain exactly investigations and notes");
if (!isObject(data.investigations)) fail("investigations must be an object");
if (!Array.isArray(data.notes) || !data.notes.every(note => typeof note === "string")) fail("notes must be an array of strings");
const keyPattern = /^(parallelism\|(wp|rr)|startup\|(so|bc)|fixtures\|(fsr|psf|ref|ss|clc)|long-tests\|ltr)\|[^|]+$/;
const outcomePattern = /^(regression|fast_no_improvement|pattern_mismatch|already_optimized|too_complex): .+/;
for (const [key, value] of Object.entries(data.investigations)) {
if (!keyPattern.test(key)) fail(`invalid investigation key: ${key}`);
if (!(Number.isInteger(value) && value > 0) && !(typeof value === "string" && outcomePattern.test(value))) {
fail(`invalid outcome for investigation: ${key}`);
}
}
console.log("ci-perf notes.json conforms to schema");
```

Other workflows need still more imperative code for nested object shapes, enums, timestamps, optional fields, and uniqueness checks.

Most of the structural constraints above are naturally expressed by JSON Schema:

```json
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"required": ["investigations", "notes"],
"additionalProperties": false,
"properties": {
"investigations": {
"type": "object",
"propertyNames": {
"pattern": "^(parallelism\\|(wp|rr)|startup\\|(so|bc)|fixtures\\|(fsr|psf|ref|ss|clc)|long-tests\\|ltr)\\|[^|]+$"
},
"additionalProperties": {
"anyOf": [
{ "type": "integer", "minimum": 1 },
{
"type": "string",
"pattern": "^(regression|fast_no_improvement|pattern_mismatch|already_optimized|too_complex): .+"
}
]
}
},
"notes": {
"type": "array",
"items": { "type": "string" }
}
}
}
```

## What can be done with the current API

A `validation.script` can write an inline schema to a temporary directory and invoke a pinned validator such as `ajv-cli`:

```yaml
validation:
timeout-minutes: 1
script: |
const { execFileSync } = require("child_process");
const os = require("os");

const schema = {
$schema: "http://json-schema.org/draft-07/schema#",
type: "object",
required: ["investigations", "notes"],
additionalProperties: false,
properties: {
investigations: {
type: "object",
propertyNames: {
pattern: String.raw`^(parallelism\|(wp|rr)|startup\|(so|bc)|fixtures\|(fsr|psf|ref|ss|clc)|long-tests\|ltr)\|[^|]+$`
},
additionalProperties: {
anyOf: [
{ type: "integer", minimum: 1 },
{
type: "string",
pattern: "^(regression|fast_no_improvement|pattern_mismatch|already_optimized|too_complex): .+"
}
]
}
},
notes: {
type: "array",
items: { type: "string" }
}
}
};

const schemaDir = fs.mkdtempSync(path.join(os.tmpdir(), "ci-perf-schema-"));
const schemaPath = path.join(schemaDir, "schema.json");

try {
fs.writeFileSync(schemaPath, JSON.stringify(schema));
execFileSync(
"npx",
[
"--yes",
"ajv-cli@5.0.0",
"validate",
"--spec=draft7",
"-s", schemaPath,
"-d", path.join(memoryRoot, "notes.json")
],
{ stdio: "inherit" }
);
} finally {
fs.rmSync(schemaDir, { recursive: true, force: true });
}
```

This is more declarative in its central section, but the adapter is substantial and introduces package download, network, startup-time, and supply-chain concerns. Validation may run more than once as part of defense in depth, multiplying those costs.

## Runner availability research

A schema validator should not be assumed to exist on GitHub-hosted Ubuntu runners:

- The [GitHub-hosted runners reference](https://docs.github.com/en/actions/reference/runners/github-hosted-runners) points to the runner-image manifests as the source of installed-software details.
- The current [`ubuntu-24.04` manifest](https://github.com/actions/runner-images/blob/main/images/ubuntu/Ubuntu2404-Readme.md) and [`ubuntu-22.04` manifest](https://github.com/actions/runner-images/blob/main/images/ubuntu/Ubuntu2204-Readme.md) list Node.js, npm, Python, pip/pipx, and `jq`, but do not list `ajv`, `ajv-cli`, `check-jsonschema`, or Python's `jsonschema` package. `jq` can implement ad hoc predicates but is not a JSON Schema validator.
- The runner-images repository's own [`CheckJsonSchema.ps1`](https://github.com/actions/runner-images/blob/main/helpers/CheckJsonSchema.ps1) installs `GripDevJsonSchemaValidator` on demand rather than relying on a preinstalled validator.
- Runner images are updated weekly, so an incidental/transitive package should not be treated as a supported runtime contract.

Therefore, `npx ajv-cli@...`, `pipx run check-jsonschema==...`, or similar is possible only as an explicit downloaded dependency. It is not a zero-dependency default-runner solution and additionally depends on the workflow network policy allowing the relevant registry.

`gh-aw` also currently ships an internal [`validateValueAgainstSchema`](https://github.com/github/gh-aw/blob/main/actions/setup/js/mcp_scripts_validation.cjs) helper. It is attractive because it is already present at runtime, but it is undocumented for memory validators and implements only a subset: types, `oneOf`/`anyOf`, `enum`, `required`, `additionalProperties: false`, properties, and array items. It does not cover important constraints in this example such as `propertyNames`, `pattern`, `minimum`, or schema-valued `additionalProperties`. Importing an internal action helper would also create unsupported coupling.

## Design options

### 1. Native inline schema (preferred)

```yaml
tools:
repo-memory:
allowed-extensions: [".json"]
validation:
json-schema:
file: notes.json
schema:
type: object
required: [investigations, notes]
additionalProperties: false
properties:
# ...
```

The compiler could validate the schema itself, embed it in the generated workflow/manifest, and use a bundled, pinned validator at runtime. This avoids registry/network dependencies and provides consistent JSON-pointer diagnostics. Existing `script:` could remain available and run after schema validation for domain-specific invariants.

**Pros:** declarative, reviewable, deterministic, consistent errors, no per-workflow validator code, no runtime download.

**Cons:** adds a bundled runtime dependency and requires choosing/documenting supported JSON Schema draft(s), formats, and resource limits.

### 2. Schema files and/or per-file mappings

```yaml
validation:
json-schemas:
notes.json: ./schemas/notes.schema.json
metrics/*.json: ./schemas/metric.schema.json
```

At compile time, resolve, validate, hash, and embed referenced schemas so validation also works in repo-memory push jobs where repository checkout/layout may differ.

**Pros:** schemas can be reused, linted, and tested independently; scales to multi-file memories.

**Cons:** path/glob semantics and missing-file behavior need specification; external `$ref` resolution can create portability and network/security issues. Restricting references to local files and embedding the resolved schema would help.

### 3. Promote/extend the existing internal validator

Expose a supported helper to `validation.script` and extend it toward the necessary JSON Schema vocabulary.

**Pros:** relatively small frontmatter/API change and reuses current runtime code.

**Cons:** workflows still need adapter code; maintaining a home-grown partial JSON Schema implementation is risky, and silently ignored unsupported keywords would be especially dangerous.

### 4. Declarative dependencies for validation scripts

Allow a validator dependency declaration, for example:

```yaml
validation:
dependencies:
npm: ["ajv@8.17.1"]
script: |
const Ajv = require("ajv");
# ...
```

**Pros:** general solution for arbitrary validation libraries and preserves script flexibility.

**Cons:** runtime downloads, registry/network dependency, repeated installation, larger attack surface, lock/integrity policy questions, and still more boilerplate than native schema support.

### 5. Bundle a generic validator utility without new frontmatter

Document a stable runtime helper that accepts a schema and value/file, backed by a mature bundled validator.

**Pros:** smaller API surface and no network dependency.

**Cons:** schema remains embedded in JavaScript rather than YAML and every workflow repeats loading/error-handling glue.

## Non-schema constraints

JSON Schema handles shape, types, patterns, ranges, required fields, and many conditional rules well. It does not naturally express all domain invariants, such as uniqueness of a projection across objects (for example, unique `(digest, route)` pairs), relationships between files, or checks requiring current time/external state. The design should therefore support composition:

1. generic storage checks;
2. declarative schema validation;
3. optional `validation.script` for remaining semantic/cross-file checks.

## Suggested direction

Add native `json-schema`/`json-schemas` support backed by a pinned validator bundled with `gh-aw`, while retaining `script` as an optional second stage. Compile-time schema validation plus runtime data validation would avoid both typo-prone imperative checks and runtime package downloads.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.