GoogleCloudPlatform / GoogleCloudPlatform/generative-ai

[Bug]: Generated autonomous artifacts produce inconsistent metrics for the same scenario

Open
#3,048 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
17.7k
Forks
4.5k
Avg merge
12h 38m
Merged PRs (30d)
42

Description

### File Name

search/gemini-enterprise/ge-demo-generator/agent_template/adk_agent/app/

### What happened?

I reproduced a quantitative consistency issue in autonomous artifacts generated by the Gemini Enterprise Demo Generator.

The generated agent can produce different numerical results for the same business scenario across HTML simulators, PDF reports, and later generated presentation decks.

I first asked the generated retail operations agent to build an interactive district inventory What-If Simulator and an executive summary PDF.

The generated HTML simulator used the following Winter Storm scenario:

Safety stock: 25%
Weather / transit delay: +4 days

The interactive simulator calculated:

Perishable Waste: USD 26,432
Stockout Risk: 23.4%

I later asked the same generated demo environment to produce a District Optimization & Waste Reduction Proposal deck using the same operational data and simulator context.

The generated PowerPoint contained the same Winter Storm scenario:

Safety stock: 25%
Transit delay: 4 days

However, the deck calculated:

Perishable Waste: USD 18,900
Stockout Risk: 16.2%

These are materially different results for the same scenario parameters.

I also observed inconsistency between the generated HTML and PDF/report narrative. Metrics and risk claims were not always sourced from one canonical calculation result before being rendered into the different deliverables.

Actual behavior:

Different generated artifacts independently produce or restate quantitative outcomes, resulting in conflicting KPI values for the same scenario.

Expected behavior:

A scenario should be calculated once using an authoritative/canonical calculation result.

All generated outputs that reference that scenario should consume the same structured result payload, for example:

scenario:
safety_stock_pct: 25
transit_delay_days: 4
perishable_waste_usd: 26432
stockout_risk_pct: 23.4

The HTML simulator, PDF report, A2UI response, and PowerPoint should render those same values rather than independently recomputing or regenerating the metrics in natural language.

If a later autonomous task intentionally uses a different calculation model or baseline, the artifact should explicitly identify the changed assumptions instead of presenting the result as the same scenario.

Why this matters:

These artifacts are intended for executive and customer-facing demos. Conflicting numbers between an interactive simulator, executive PDF, and proposal deck significantly reduce trust in the generated outputs.

This is also not solved by manually correcting one generated demo artifact. If the artifact-generation workflow does not preserve canonical calculated values, newly generated demos can reproduce the same inconsistency.

I validated the issue by comparing the actual generated HTML simulator values with the generated proposal deck produced from the same demo environment and operational dataset.

Reported and reproduced by:
Nilden Tutalar
Cloud Solution Architect - Cloud4Next
Contact: nilden.tutalar@cloud4next.com

### Relevant log output

```shell
Generated What-If Simulator scenario:

Scenario: Winter Storm
Safety stock: 25%
Transit / weather delay: +4 days

HTML simulator result:
Perishable Waste: USD 26,432
Stockout Risk: 23.4%

Later generated District Optimization Proposal deck:

Scenario: Winter Storm
Safety stock: 25%
Transit delay: 4 days

PowerPoint result:
Perishable Waste: USD 18,900
Stockout Risk: 16.2%

Same scenario parameters:
25% safety stock
4-day delay

Different generated outcomes:
Waste delta: USD 7,532
Stockout-risk delta: 7.2 percentage points

The artifacts were produced in the same generated demo environment using the same operational dataset.
```

### Code of Conduct

- [x] I agree to follow this project's Code of Conduct

Contributor guide

Open the contributing guide

Research direction

Start in search/gemini-enterprise/ge-demo-generator/agent_template/adk_agent/app/ and trace how the HTML simulator, PDF report, A2UI response, and PowerPoint deck receive scenario metrics. Reproduce the Winter Storm case and compare the values passed to each artifact. Done means one authoritative structured result is reused consistently, or changed assumptions are explicitly identified.

Written by the indexing model from the issue text.

Assessment

Domain
ai
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.