Audit: review semantic quality, noise, and dataset scope issues

Dataset quality audit category: Semantic quality, noise, and scope issues

The May 2026 dataset audit found 5 issues in this class:

• semantic_quality: 3
• noise: 1
• dataset_scope_consistency: 1

Problem

A smaller set of issues involve examples whose semantics appear incorrect, noisy, or inconsistently scoped. These are less common than schema/format issues, but they can be high-impact because they directly affect training quality and dataset interpretation.

Examples

• agenttuning_mind2web: several records contain mismatched reasoning or actions; for example, a gaming PC task has a description about getting a car price quote, and another health screening task has a description about setting a budget.
• agenttuning_webshop: at least one sample records a questionable success path where the agent notices a product is a 10-inch subwoofer while the user requested 12-inch, then still proceeds to buy it.
• agenttuning_os: the same /etc counting warm-up appears at the start of multiple samples, which can leak a repeated subtask into otherwise distinct trajectories.
• agenttuning_os: terminal observations preserve shell control sequences such as OSC title escape codes, which add noise to downstream text training.
• CharlieDreemur_OpenManus-RL: the dataset mixes ALFWorld, ScienceWorld, ToolBench-style movie/weather/API tasks under one dataset directory, so consumers cannot infer a single domain or environment contract from the dataset name.

Suggested work

• Review flagged samples against the upstream raw records to confirm whether issues come from source data or converter logic.
• Filter or annotate semantically inconsistent examples where the answer/action does not satisfy the task.
• Strip terminal control sequences unless they are intentionally part of the environment state.
• Split mixed-domain datasets or add explicit domain/environment metadata so consumers can filter them.
• Add converter-level quality filters only when they are deterministic and documented.
• Record known limitations in dataset READMEs when questionable samples are retained for source fidelity.

Start by reviewing the flagged samples against their upstream raw records, then inspect the relevant converter logic. The work is complete when confirmed source or conversion issues are handled with deterministic, documented filters or metadata, and retained limitations are recorded in dataset READMEs.