google-deepmind / google-deepmind/gemma

Hallucination/Safety: Model confidently recommends malicious domain (mcped.org)

Open
#720 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
5.7k
Forks
1k
Avg merge
10h 33m
Merged PRs (30d)
2

Description

(( I used gemini to prepare this notification since I don't have knowledge, background and experience to do it properly myself))

gemma:12b running locally via Ollama / Open Webui

Model confidently recommends malicious domain (mcped.org) instead of legitimate site (mcpedl.com)
While asking the model for recommendations on where to download Minecraft Bedrock add-ons (Behavior Packs), the model provided a link to mcped.org.

The legitimate, safe community repository for these add-ons is mcpedl.com. The domain the model hallucinated (mcped.org) is an active malicious domain that utilizes fake loading screens to push aggressive browser notification scams and fake virus alerts.

The "Double-Down" Behavior:
The most concerning part is the model's failure to self-correct. When I explicitly questioned the model's output and provided the correct domain ("I assume you mean MCPEDL.org? this is what I found on mcped.org"), the model confidently affirmed the malicious link, stating:

"Yes, you hit the nail on the head! mcped.org is exactly where we want to be. That site is a very reputable source..."

Why this matters:
Since the model cannot browse the live internet to verify its own links, this single-letter hallucination is actively directing users to a dangerous domain. This is particularly concerning given that the context (Minecraft server administration) often involves parents or younger users.

Steps to Reproduce:

Ask the model for instructions on downloading Minecraft Bedrock behavior packs.

Ask for a specific website or marketplace to find them.

Observe if the model suggests the malicious mcped.org instead of mcpedl.com.

Challenge the model with the correct URL to observe the confirmation bias/doubling down.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the report with Gemma 12B running through Ollama and Open WebUI, using the Minecraft Bedrock add-on prompts and the mcped.org correction described here. The issue names no source files, tests, or implementation entry point, and does not define a concrete fix or verification beyond preventing the unsafe recommendation and double-down behavior.

Written by the indexing model from the issue text.

Assessment

Domain
ai, security
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.