Back to Blog
•By Sumit Pandey, PhD•2 min read

ML 10: The Goldfish (RNNs)

Giving computers a memory (even if it's a short one).

Sequence Matters

Normal Neural Nets have Amnesia. When they look at Input #2, they have completely forgotten Input #1. This is fine for images (a cat is a cat, regardless of the previous photo).

But usually, order matters.

  • "I love..." -> (Next word likely: "you", "pizza", "lamp").
  • "I hate..." -> (Next word likely: "spiders", "traffic").

You need Context. You need Memory.

The Recurrent Neural Network (RNN)

An RNN is a neural network that loops back on itself. When it processes a word, it passes a little "state" (memory) to the next step. "Okay, the last word was 'Love', so keep that in mind when guessing the next one."

The Goldfish Memory

A toy memory model: add words and watch older context fade.

Hidden state · memory
Add a word to start building a memory.
Next word prediction:
...
Fresh
Fading
Forgetting

Illustrative memory decay with built-in word pairs; this is not a trained RNN.

Explore the full playground →

The Vanishing Gradient (The Goldfish Problem)

Basic RNNs are dumb. They have very short-term memory. By the time they get to the end of a long sentence, they forgot the beginning. "The cat, which was eating the food on the table near the window... was happy." The RNN forgets "cat" and gets confused about "was" vs "were".

The Goldfish Memory

LSTM and GRU (The Elephants)

To fix this, we invented LSTMs (Long Short-Term Memory). They have gates:

  1. Forget Gate: What info is trash? Throw it out.
  2. Input Gate: What info is new and important? Keep it.
  3. Output Gate: What should I pass to the next step?

This lets them remember things for much longer.

Summary

RNNs allow AI to read text, understand speech, and predict stock prices (badly). But they are slow. They read one word at a time.

Next up: The modern King of AI that killed the RNN.

Share this article