CNNs and RNNs for Text

#The Problem

TF-IDF and Word2Vec produced flat vectors that ignored word order. A classifier built on them could not tell dog bites man from man bites dog. Word order sometimes carries the signal.

Two families of architectures filled that gap before transformers arrived.

Convolutional nets for text (TextCNN). Apply 1D convolutions over sequences of word embeddings. A filter of width 3 is a learnable trigram detector: it spans three words and outputs a score. Stack different widths (2, 3, 4, 5) to detect multi-scale patterns. Max-pool to a fixed-size representation. Flat, parallel, fast.

Recurrent nets (RNN, LSTM, GRU). Process tokens one at a time, maintaining a hidden state that carries information forward. Sequential, memory-bearing, flexible input lengths. Dominated sequence modeling from 2014 to 2017, then attention happened.

This lesson builds both, then names the failure that motivated attention.