stacklok / stacklok/mecatl

Add concept of FACTS.md or canned, optimized tool calls

Open
#783 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
152
Forks
16
Avg merge
14h 48m
Merged PRs (30d)
536

Description

Problem: no matter how much you prompt in the context, agents will re-verify facts anyway. Worse yet, the few times that an agent does take a statement at face value, it may actually be wrong.

Idea for a solution:

  • Create the concept of FACTS.md or similar
  • This file contains factual statements coupled with verification script that can be automatically evaluated
  • When loading FACTS.md into contex, Mecatl automatically evaluates them, and only the facts that actually stand get loaded into context
  • The script that was ran, and it's stdout are also loaded into context, as if the agent had itself made the same those tool calls itself
  • There is a strict character budget for verifications, and the number of facts is also limited -> incentive to use it only to improve the speed and token economics of flows that the agent would otherwise always evaluate using tool calls

Potential further improvements:

  • Facts themselves can have trigger conditions, for instance certain regular expressions matching a tool call or a message -> the agent could receive a canned answer from a more efficient tool call rather than a multi-step expression
  • A cheap model could actually classify tool calls automatically or even write rewrite rules automatically
  • There could be a way to reject CI when some facts become outdated

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names no files, tests, or entry points, so begin by locating how Mecatl loads context and executes tool calls. Review the proposed FACTS.md format, verification output, character and fact limits, and trigger conditions; the work is done when the project has an agreed implementation scope and tests for accepted and rejected facts.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
ai-infra-agents
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.