OpenAI's CLIP flipped the script on computer vision by tossing out fixed label sets and instead training on 400 million (image, text) pairs scraped from the internet. Part 2 of this three-part review digs deeper into the mechanics that make contrastive image-text pretraining tick, and why matching a caption to an image turns out to be a shockingly efficient path to SOTA visual representations. For anyone building multimodal systems where speech, text, and vision now share embedding space, the design lessons here echo directly into the audio world (think CLAP for audio-text retrieval, Whisper-style weak supervision, and the speech-LLM alignment work now landing at ICASSP and Interspeech).

This installment walks through how CLIP transfers zero-shot to 30+ downstream benchmarks (OCR, action recognition, geo-localization, fine-grained classification) and hits ResNet-50-on-ImageNet accuracy without touching the 1.28M training images. The discussion frames why natural language supervision scales where curated labels stall, and how that same principle is now reshaping how we pretrain foundation models across modalities. If you are thinking about weakly-supervised pretraining for ASR or TTS, this middle chapter is where the payoff lands, so queue it up when you have twenty focused minutes.