future-agi / future-agi/future-agi

model_hub: populate_placeholders silently passes unresolved {{tokens}} to LLM on column resolution failure

Open
#321 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
2k
Forks
628
Avg merge
1d 50m
Merged PRs (30d)
167

Description

Summary

populate_placeholders() catches all non-media exceptions and returns the original (unsubstituted) messages as a fallback. Unresolved {{column_name}} tokens reach the LLM verbatim with no error surfaced to the caller.

File: futureagi/model_hub/views/run_prompt.py:460-467

    except Exception as e:
        if media_error:
            raise e
        else:
            traceback.print_exc()
            logger.exception(f"Fatal error processing messages: {e}")
            # Return original messages as fallback
            return messages

The inner per-column loop (lines 403-407) also continues silently on column resolution errors.

Impact

  • LLM receives {{column_name}} as literal text when a column is missing, a cell is null, or column lookup fails.
  • The prompt response is subtly wrong with no error surfaced to the user.
  • Callers cannot distinguish "all resolved" from "fallback due to exception" — both return a list of messages.
  • Parallel to simulate issue #312.

Suggested fix

After template rendering, scan for remaining {{...}} tokens in the output messages. If any are found, either:

  • Return an error status (preferred), or
  • Log at WARNING with the list of unresolved tokens.

See also

ADR 024 in docs/adr/024-populate-placeholders-no-unresolved-validation.md.
Simulate issue #312 for the parallel problem in the simulate module.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in futureagi/model_hub/views/run_prompt.py at populate_placeholders(), especially the per-column loop around lines 403-407 and fallback around lines 460-467. Read docs/adr/024-populate-placeholders-no-unresolved-validation.md and compare the parallel simulate issue #312. Done means unresolved {{...}} tokens cannot reach the LLM silently and the caller receives or can identify the failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.