Learning neural networks through mental models

For a while, I understood neural networks mostly mechanically. Data entered the network, passed through layers, and eventually produced a prediction. Loss was calculated, gradients computed and weights updated. I could follow the sequence but the pieces still felt strangely arbitrary. Why this structure? Why did stacking layers make a network so powerful? Why were activation functions so important? I understood much of what the network was doing, step by step, without feeling like I understood the network as a whole.

Then I watched Josh Starmer’s StatQuest video on the key ideas underlying neural networks. He presented neural networks as systems that sculpt functions. Instead of following data through layers, I could zoom out and picture what those transformations were collectively constructing. That shift helped immediately.

Then I got greedy and started looking for other ways to think about neural networks. By the time I was done, I realised that different mental models made different questions easier to answer. Some let me zoom out and collapse the network into one idea. Others did the opposite, breaking it into smaller transformations or computations. Some connected neural networks to ideas I already understood; others made me reinterpret things I thought I understood already. Together, they made neural networks feel much less mysterious. Learning to switch between these mental models was the breakthrough I needed.

Neural networks as stacks of transformations

The mechanical understanding I started with was essentially viewing neural networks as a stack of transformations.

Each layer progressively transforms the data before passing it on:

where

x is the input
h’s are intermediate hidden representations, and
y is the final output.

This was probably my default mental model before I knew to call it one. It made the forward pass easy to follow: the input is transformed step by step until the network produces an output.

This perspective helps explain why neural networks have depth at all. Each layer can transform the output of the previous layer, allowing the network to build increasingly complex computations. Earlier layers produce simpler transformations that later layers can refine further.

At the same time, this perspective kept my attention on the individual steps. I didn’t have a clear picture of what all those transformations were collectively doing. I needed to zoom out.

Neural networks as function approximators

Function approximation gave me that zoomed-out view.

I was already comfortable thinking about models like linear and logistic regression as functions. Given some inputs, these models learn a mapping that produces an output. A neural network can be understood in exactly the same way:

This stripped away much of the apparent mystery surrounding neural networks. Underneath all that machinery, the network was still doing something familiar: learning a function that maps inputs to outputs, just like the other models I already understood. The difference was that neural networks could learn far more flexible and expressive functions.

But how do these complex functions emerge from simple neurons?

Neural networks as function sculptors

I already understood abstractly that stacking layers with nonlinear activation functions allowed neural networks to model increasingly complex relationships. But I couldn’t visualise it.

In Josh Starmer’s explanation, each neuron works with a given activation function. The weights and biases slice, flip and stretch that same activation function into new shapes. As a result, each neuron applies a small nonlinear transformation to its input. These small transformations and shapes are stitched together across layers to create yet new shapes. A complex decision boundary emerges from many local transformations.

Rather than learning one giant function directly, neural networks build complex behaviour by combining many smaller nonlinear transformations.

With Softplus, a neuron can contribute a smooth bend into the function surface. With ReLU, neurons contribute piecewise linear changes. Deep networks combine huge numbers of these simple transformations, progressively sculpting a much more complex function.

Don’t take my word for it. Watch the video. His animations (and the noises he makes while calculating) make it worth your time.

What clicked for me was that the network didn’t need any individual neuron to represent a complicated function. Each neuron could contribute something simple. Complexity emerged from composing and combining many of those simple transformations.

Function approximation had helped me zoom out and see the network as one flexible function. Function sculpting let me zoom in just enough to see how that flexibility could be constructed.

Neural networks as representation learners

One thing that distinguishes neural networks from many of the models I was used to working with is that they can also learn how to represent the input itself.

In my previous experience with econometrics and causal inference, deciding how to represent a problem happened largely before fitting the model. I thought carefully about which variables to include, what they measured, and how they should be transformed. The model then learned from the representation I had given it.

With neural networks, deciding how to represent the problem doesn’t happen entirely before training. The network also learns useful ways of representing the data. The hidden states in the stack of transformations aren’t just intermediate values but intermediate representations shaped during training to become useful for the task. This is why neural networks are often described as systems for representation learning.

This applies even to relatively simple neural networks. For example, a churn model might learn hidden behavioural patterns that help distinguish customers with different risks of leaving. In a larger language model, the learned representations can encode complex semantic relationships. The scale and complexity differ but both are forms of representation learning.

I had been thinking of learning primarily as learning a mapping from my representation of the problem to the target. Representation learning made me realise that the representation itself could be part of what was learned.

Neural networks as optimisation systems

One thing that puzzled me when learning about neural networks was the amount of attention given to activation functions. I understood that without them, stacking linear layers still collapses into a linear transformation. But why did the choice of activation function matter?

I initially thought about activation functions through the function-sculpting perspective. Different activation functions give the network different shapes to work with, so I assumed that choosing between sigmoid, tanh or ReLU was primarily about the kinds of functions the network could construct.

The optimisation perspective gave me a different answer.

Unlike simpler models such as linear regression, neural networks generally do not have neat closed-form solutions for their parameters. These parameters are learned iteratively through optimisation. At first glance, optimisation can seem like a secondary implementation detail — merely the mechanism through which weights get updated during training.

But it isn’t enough for a neural network to be capable of representing a useful function; optimisation has to find the parameters that produce one. Neural networks were already highly expressive decades ago. The challenge was whether optimisation could actually find a useful solution. Gradient-based optimisation struggled to reliably train deep networks. Much of the progress in deep learning has involved making highly expressive neural networks easier to train.

Many neural network design choices are intended to solve optimisation problems. Sigmoid activation functions can saturate, while ReLU often allows gradients to propagate more effectively. Weight initialisation affects whether signals and gradients remain stable across many layers. Learning rates and optimisers determine how we move through the loss landscape.

This perspective also changed how I thought about architecture. From my original stack-of-transformations perspective, residual connections seemed strange. If each layer was supposed to progressively transform the representation, why let an earlier representation bypass a block and add it back later?

From an optimisation perspective, the question changes. Residual connections provide identity paths through deep networks, helping information and gradients flow and allowing blocks to learn changes to an existing representation. What had looked awkward from one mental model made much more sense from another.

This perspective helped explain why so much of deep learning is about more than designing expressive architectures. It is also about making those architectures trainable.

Neural networks as computational graphs

The optimisation perspective explains why gradients matter. But how are the gradients computed?

Function approximation had taught me to think of a neural network as one enormous function. But trying to imagine its derivatives gave me a headache. Differentiating an enormous, complicated function with millions of parameters sounded overwhelming.

The computational graph perspective reverses that. It breaks this gigantic function back into the small operations that produce it.

During the forward pass, data moves through the graph step by step: inputs are transformed; activations are computed; predictions are produced; and loss is calculated. Backpropagation then works backwards through this graph to compute the gradients.

The computational graph perspective decomposes a neural network into smaller differentiable computations. During the forward pass, values flow through the graph to produce a prediction and compute the loss. During the backward pass, gradients flow back through the same graph using the chain rule.

The key insight is that the gradients aren’t computed as one enormous derivative. Each operation only needs to compute local derivatives with respect to its own inputs. By the magic of chain rule, these local derivatives are stitched together, and gradients are able to propagate backward through the whole network. Backpropagation no longer felt like a memorised algorithm.

Conclusion

Neural networks became much easier to understand once I stopped searching for a single explanation.

What I initially thought of as one complicated system could be viewed at very different scales. I could zoom out and view the system as a single function, or zoom in and think about individual transformations and computations. I could think about what representations the network was learning, or switch questions entirely and ask whether optimisation could actually find them.

What surprised me most was that a mental model that made one question easier could make another harder to reason about.

Thinking of a neural network as one enormous function connected it to more familiar machine learning models but made the idea of differentiating it overwhelming. Instead, breaking that same function into a computational graph made backpropagation much easier to understand.
Residual connections were initially confusing when I viewed networks as stacks of transformations. However, from an optimisation perspective, their purpose made more sense.

No single perspective fully explains neural networks. Each lens makes different questions easier to reason about. The real shift was learning which mental model to reach for when something stopped making sense.

The mental models: a summary

Mental Model	Question	Main Insight
Stack of Transformations	Why are there multiple layers?	Layers progressively reshape data into new representations.
Function Approximation	What is the network fundamentally doing?	A neural network is ultimately learning a function from inputs to outputs.
Function Sculpting	How do simple neurons create complex functions?	Complex functions emerge from many simple nonlinear transformations.
Representation Learning	What are hidden layers learning?	Neural networks learn useful internal representations of the input.
Optimisation System	Why do so many design choices concern training?	Expressiveness isn’t enough; optimisation has to find useful parameters.
Computational Graph	How are gradients actually computed?	Derivative calculations can be decomposed into local derivatives stitched together by the chain rule.