Olewave - Professional and Trustworthy - Data Services and Solutions

Tutorials

[Long Review] Fully Sharded Data Parallel: faster AI training with fewer GPUs

Training a billion-parameter speech or language model on a modest GPU budget used to mean picking your poison across pipeline parallelism, tensor parallelism, …

Apr 23, 2022· 1 min read

Tutorials

[Short Review] Fully Sharded Data Parallel: faster AI training with fewer GPUs

Fully Sharded Data Parallel is Meta's answer to a question every speech and language team eventually asks: how do we train models an order of magnitude bigger …

Apr 23, 2022· 1 min read

Tutorials

[Long Review] Deduplicating Training Data Makes Language Models Better

Deduplication sounds boring until you learn that a single 61-word sentence appears more than 60,000 times inside C4, and that language models trained on such c…

Apr 16, 2022· 1 min read