Enhancing DiLoCo robustness: Addressing model poisoning via Semantic Gating

Hi DeepMind Team,

I’ve been studying your work on DiLoCo (Distributed Low-Communication) and its efficiency in high-latency environments. It's a game-changer for decentralized training.

However, during my benchmarks, I identified a significant vulnerability: the architecture is highly susceptible to model poisoning attacks during the local SGD phases. Since global synchronization is sparse, a single malicious actor can inject significant drift that bypasses standard numerical aggregators.

I’ve developed a defense layer called Semantic Guard that uses 32-dimensional latent representations (Semantic Atoms) to validate the "intent" of local updates before global aggregation. This ensures that only semantically consistent updates are merged.

Key Result:
In a simulation with 40% malicious nodes, the standard DiLoCo baseline collapsed (~50% accuracy), while the Semantic Guard maintained 98% accuracy.

I’ve open-sourced a reference implementation and the results here:
🔗 https://github.com/LelloOmwei/semantic-guard-diloco

I’d love to hear your thoughts on integrating such semantic validation layers into future iterations of asyncdiloco to enhance security in untrusted decentralized setups.