Scaling a Transformer past a hundred billion parameters gets philosophical fast: do you activate the whole network for every token, or route each token to the specialists that actually need to see it? GShard is Google's landmark answer, pairing sparse Mixture-of-Experts layers with an automatic sharding annotation system that lets a single-program, multiple-data XLA compiler place a 600-billion-parameter multilingual translation model across thousands of TPU cores.

This long review walks through the top-2 gating, the auxiliary load-balancing losses, and the annotation-driven parallelism that makes GShard tick, plus the translation quality wins that put MoE architectures on the industry roadmap. For voice AI teams eyeing MoE speech foundation models or multilingual ASR that scales without linear compute blow-up, GShard is the paper that set the vocabulary later borrowed by Switch Transformer, GLaM, and modern speech MoE work. Block out an hour if sparse scaling is anywhere in your architecture future.