harness / harness/cli

create: empty mutable-field registry breaks --set for many nouns (all AI-Evals, environment, service, repository) with no -f fallback

Open
#28 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
22
Forks
10
Avg merge
14h 28m
Merged PRs (30d)
54

Description

Summary

For many create nouns, the --set key=value flag rejects every field with unknown or read-only field "<field>", even though --help advertises those exact fields. --list-fields confirms the cause: these nouns return "No mutable fields defined for this command" — the mutable-field registry is empty, so no --set value is ever accepted. There is also no -f/--file fallback, leaving these nouns with no working create path via the CLI at all.

This appears to be the same root cause as #27 (connector) and #23 (PR), but the scope is much broader — it affects all AI-Evals nouns plus several core ones.

Version

harness version 0.1.10 (2026-06-18T04:38:52Z)

Reproduction

$ harness create eval_target --org default --project <proj> \
    --set name=repro --set identifier=repro --set type=prompt
unknown or read-only field "type"; use --list-fields to see mutable fields

$ harness create eval_target --list-fields
No mutable fields defined for this command.

--help for the same command advertises the fields it then rejects:

$ harness create eval_target --help
Create an AI Evals target: harness create eval_target --set name=... identifier=... type=prompt

Scope — --list-fields audit

Working (registry populated):

Noun --list-fields
secret ID / Label / Type ✅
project ID / Label / Type ✅
organization ID / Label / Type ✅

Broken (empty registry → --set unusable, no -f fallback):

Noun --list-fields
eval_target No mutable fields ❌
eval_dataset No mutable fields ❌
eval_metric No mutable fields ❌
eval_model No mutable fields ❌
eval_metric_set No mutable fields ❌
evaluation No mutable fields ❌
pr No mutable fields ❌ (see #23)
connector No mutable fields ❌ (see #27)
environment No mutable fields ❌
service No mutable fields ❌
repository No mutable fields ❌

No workaround

The broken nouns also reject -f:

$ harness create eval_target -f /dev/null
unknown shorthand flag: 'f' in -f

So affected resources cannot be created through the CLI by any means; callers must drop to the REST API.

Expected

Either:

  1. Populate the mutable-field registry for these nouns so the --set fields shown in --help actually work (and support nested keys like git_source.file_path for git-backed eval resources), or
  2. Add a -f/--file <yaml|json> body flag as a general create path for complex/nested resources.

Impact

Standing up an AI-Evals stack (dataset → target → metric set → evaluation) or any git-backed eval is impossible via the CLI today — every component create fails. Same for code pr, connector, environment, service, and repository create.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the create command's --set, --list-fields, and -f/--file handling, comparing the mutable-field registry for working nouns with eval_target and the other listed broken nouns. Done means the advertised fields are accepted for affected resources, including nested git_source.file_path where applicable, or a general file-based create path is available.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
cli, developer-experience
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.