Training a billion-parameter speech or language model on a modest GPU budget used to mean picking your poison across pipeline parallelism, tensor parallelism, or ZeRO-style optimizer sharding. Facebook's Fully Sharded Data Parallel (FSDP), now shipped in FairScale and later upstreamed to PyTorch, offers a cleaner answer: shard the parameters themselves across data-parallel workers, gather them just-in-time for each microbatch, and overlap communication with computation so the wall-clock cost stays low.

This long review walks through why FSDP is conceptually simpler than intra-layer or pipeline parallelism, how uniform parameter sharding beats optimizer-plus-gradient-only approaches, and how optional CPU offload lets you fit models that shouldn't fit. If you're training TTS acoustic models, wav2vec-style foundation models, or LLMs for voice assistants, FSDP is quickly becoming table stakes for scaling without a data-center-sized cluster. Settle in for the deep-dive if you want the mechanics before you flip the switch in your own training script.