Neural Networks — How AI Works

The Hook: Your Brain Didn't Invent This

Everyone talks about neural networks like they're inspired by the brain. That's true, technically. But it's also misleading. What you're actually looking at is a very specific piece of math that happened to work extremely well on a very specific class of problems.

The brain analogy is mostly for marketing. The real story is: calculus + linear algebra + lots of data = useful pattern recognition. That's it. That's the whole thing.

Key Insight

A neural network is a function. A big, messy, parameterized function. You feed it numbers, it spits out other numbers. The "learning" is just adjusting the parameters so the right inputs produce the right outputs.

What Is a Neuron?

Start simple. A single neuron takes a list of numbers, multiplies each by a weight, sums them up, adds a bias, then pushes the result through a nonlinear activation function.

That's it. That's the neuron.

// A single neuron, mathematically
output = activation(
    w1 * x1 + w2 * x2 + w3 * x3 + ... + wn * xn + bias
)

// activation function examples (nonlinearities):
// ReLU: max(0, x)
// Sigmoid: 1 / (1 + e^(-x))
// Tanh: (e^x - e^(-x)) / (e^x + e^(-x))
    

The weights and bias are what get "learned." Initially they're random. Through a process called backpropagation (which we'll get to), the network tweaks them step by step to minimize how wrong it is.

Layers: Stacking Neurons

One neuron isn't useful. Stack hundreds or thousands into layers, and you get a network that can learn complex patterns. Data flows left to right: input layer → hidden layers → output layer.

INPUT HIDDEN LAYER OUTPUT ○ ○ ○ x₁ ────────────────○─────────────────────○ ╱ ╲ │ ○ ○ ─ ○ ─○ ○ x₂ ────────────────○─────────────────────○ ╲ ╱ │ ○ ○ ─ ○ ─○ ○ x₃ ────────────────○─────────────────────○ Each arrow = a weight. Each ○ = one neuron. The magic: backpropagation tunes every weight.

The "hidden" layers are where it gets interesting. These are the layers that aren't directly connected to input or output. They're learned internal representations. In practice, these layers learn to detect features—edges, textures, patterns, concepts—without being explicitly told to.

How It Learns: Backpropagation

This is where most explanations get fuzzy. Here's the actual process:

1. Forward pass: Run input data through the network. Get an output.

2. Compute loss: Compare the output to the correct answer. Measure the error.

3. Backward pass: Propagate that error back through the network. Calculate how much each weight contributed to the error.

4. Update weights: Adjust each weight slightly in the direction that reduces the error.

Repeat this millions of times on millions of examples, and the weights converge to values that produce correct outputs. That's "learning."

// Gradient descent update rule (simplified)
for each weight w in network:
    w = w - learning_rate * gradient_of_loss_wrt_w

// The gradient tells us: "which direction does this weight need to move"
// The learning rate controls: "how much do we move it"
// Set it too high: overshoot, diverge
// Set it too low: learn painfully slow
    

Why Nonlinearities Matter

Without activation functions, stacking layers is pointless. Every layer would just be a linear transformation of the previous one, and the whole stack collapses into a single linear function. Nonlinearities (ReLU, sigmoid, etc.) let the network learn non-linear decision boundaries. Without them, you couldn't model XOR. With them, you can model anything.

Deep Networks: Why More Layers Beat More Neurons

A wide network (many neurons in one layer) can learn any function. A deep network (many layers) learns it more efficiently. Depth hierarchies work because each layer builds on the previous: edges → textures → shapes → objects → concepts. This is why deep networks beat shallow ones on perception tasks.

The Key Takeaway

A neural network is a composition of linear transforms and nonlinearities. It learns by adjusting weights via gradient descent. More layers = learned hierarchies = better pattern recognition on complex data. That's not magic. That's math that scaled.

← Back to Learn Tokenization →