Why choose between contrastive learning and masked language modeling for speech pretraining when you can wire both into the same network? W2v-BERT, out of Google, stacks a wav2vec-style contrastive module on top of a BERT-style MLM module and trains them end-to-end, so the discrete codebook learned by contrastive loss becomes the prediction target for the MLM head — no offline clustering, no two-stage pipeline.

That single-loop design turns out to be more than a tidy engineering choice. On Librispeech test-other, W2v-BERT hits a new low WER at the time of publication and generalizes cleanly to Google's massive multilingual voice search data, outperforming wav2vec 2.0 and HuBERT at comparable parameter counts. The talk breaks down the two loss terms, why joint training stabilizes codebook usage, and where the model still leaves headroom for downstream fine-tuning. If you're evaluating which self-supervised backbone to build your production ASR on, this one deserves 10 minutes of your attention.