Olewave - Professional and Trustworthy - Data Services and Solutions

Tutorials

[Olewave's Long Review] Efficient Training of Neural Transducer for Speech Recognition

Neural transducers are the workhorse of streaming ASR, but training them efficiently is famously painful: the RNN-T loss has cubic memory in sequence length, t…

Jul 23, 2022· 1 min read

Tutorials

Boris Johnson’s Rise and Fall - an analysis of the mics

Political news is not usually what voice AI engineers cue up for a Friday afternoon, but Boris Johnson's resignation announcement, delivered in front of a bank…

Jul 9, 2022· 1 min read

Tutorials

Dr. Fauci was caught on hot mic - 'what a moron' -- analysis from a research perspective

Dr.

Jul 2, 2022· 1 min read

Tutorials

How Does the All-New Dictation in iOS 16 Work? Reveal Apple's Secret Sauce by a Speech Researcher!

Apple's iOS 16 dictation runs fully on-device, and that single design choice has cascading consequences for latency, privacy, and what kinds of ASR architectur…

Jun 18, 2022· 1 min read

Tutorials

[Long Review] Conformer: Convolution-augmented Transformer for Speech Recognition

Conformer is the model that quietly took over end-to-end ASR, and its recipe, bolt a convolution module onto every Transformer block, turned out to be one of t…

Jun 11, 2022· 1 min read

Tutorials

[Long Review] Axial Attention in Multidimensional Transformers

Full self-attention over an image or a video is prohibitively expensive, so how do you keep the expressiveness of a Transformer without the quadratic blowup?

May 28, 2022· 1 min read

Tutorials

[Short Review] Axial Attention in Multidimensional Transformers

Axial attention is one of those elegant ideas that keeps Transformers tractable on high-dimensional data: instead of attending over every pixel or every time-f…

May 28, 2022· 1 min read

Tutorials

[Long Review] Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis

This is the paper that made zero-shot voice cloning practical: train a speaker encoder on a discriminative verification task with thousands of noisy, transcrip…

May 21, 2022· 1 min read

Tutorials

[Short Review] Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis

The Google paper that kicked off the modern zero-shot voice cloning wave has a beautifully clean architecture: a speaker encoder trained on verification, a Tac…

May 21, 2022· 1 min read

Tutorials

[Short Review] Wav2Seq: Pre-training Speech-to-Text Encoder-Decoder Models Using Pseudo Languages

Self-supervised pretraining like wav2vec 2.0 gave us great speech encoders, but if you want a full encoder-decoder ASR system, the decoder was still starting f…

May 14, 2022· 1 min read

Tutorials

[Long Review] Towards Zero-Label Language Learning

What if you could train task-specific NLP models without a single human-labeled example? "Towards Zero-Label Language Learning" leans hard into that question.

May 7, 2022· 1 min read

Tutorials

[Long Review] Fully Sharded Data Parallel: faster AI training with fewer GPUs

Training a billion-parameter speech or language model on a modest GPU budget used to mean picking your poison across pipeline parallelism, tensor parallelism, …

Apr 23, 2022· 1 min read

Tutorials

[Short Review] Fully Sharded Data Parallel: faster AI training with fewer GPUs

Fully Sharded Data Parallel is Meta's answer to a question every speech and language team eventually asks: how do we train models an order of magnitude bigger …

Apr 23, 2022· 1 min read

Tutorials

[Short Review] Deduplicating Training Data Makes Language Models Better

How much of your language model's "generalization" is actually just regurgitation?

Apr 16, 2022· 1 min read

Tutorials

[Long Review] Hurdles to Progress in Long Form Question Answering

Long-form question answering is one of those benchmarks that looks impressive on the leaderboard and disturbing under a microscope.

Apr 2, 2022· 1 min read

Tutorials

[Long Review] Finetuned Language Models Are Zero-Shot Learners

Before InstructGPT made instruction tuning a household phrase, Google's FLAN paper made the case that a modest amount of multi-task fine-tuning on natural-lang…

Mar 5, 2022· 1 min read

Tutorials

[Long Review] 'GShard': Scaling Giant Models with Conditional Computation and Automatic Sharding

Scaling a Transformer past a hundred billion parameters gets philosophical fast: do you activate the whole network for every token, or route each token to the …

Feb 26, 2022· 1 min read

Tutorials

Triplets-like Russian Figure Skaters: Can Kullback-Leibler Divergence be Used Tell Their Difference?

Speaker adaptation is one of those ASR problems that sounds solved until you actually try to adapt a sequence-to-sequence model without wrecking the encoder's …

Feb 19, 2022· 1 min read

Tutorials

Is Nathan Chen's 4 Flip scored by Mixture-of-Experts? Part 1: Switch Transformers: sparse MoE models

Framed with a wink at Nathan Chen's quad flip scoring, this talk unpacks Switch Transformer, Google's clean and aggressively simplified take on Mixture-of-Expe…

Feb 12, 2022· 1 min read

Tutorials

Is Nathan Chen's 4 Flip scored by Mixture-of-Experts? Part 2: GLaM:Efficient Scaling of LMs with MoE

Part 2 of the MoE mini-series turns to GLaM, Google's 1.2-trillion-parameter Mixture-of-Experts language model that matches or beats GPT-3 on zero-, one-, and …

Feb 12, 2022· 1 min read

Tutorials

BERT Paper Reviewed from a Speech Perspective

BERT hardly needs an introduction in NLP, but reading it through a speech engineer's eyes uncovers a different set of lessons than the standard "masked languag…

Feb 6, 2022· 1 min read

Tutorials

Transformer: Attention is All You Need and Listen, Attend and Spell -- from a Speech Perspective

Two papers, one story. "Attention Is All You Need" gave the world the Transformer and reset how sequence modeling gets done, while "Listen, Attend and Spell" (…

Jan 29, 2022· 1 min read