Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

Our goal is to train a binary classifier that is able to separate the two curves,
drawing a decision boundary between them. In Chapter 3, we figured out that the
decision boundary for a binary classification problem was a straight line.
So I ask you: Is it possible to draw a straight line that separates the parabolas?
Obviously not—but does it mean the problem is unsolvable? Same answer:
Obviously not. It only means we need to look at the problem from a different
perspective!
Transformations
In the "Are My Data Points Separable?" section of Chapter 3, we talked briefly
about dimensionality, the kernel trick in Support Vector Machines, and the
separability of data points. In a way, that was a different perspective already. There,
we would transform the feature space, mapping it into a higher-dimensional one,
and hoping to more easily separate the data points.
In Chapter 4, we established that, without activation functions, a deeper model
has an equivalent shallow model (a logistic regression, in case of a binary
classification). This means we need an activation function to be able to effectively
increase dimensionality and, more important, to twist and turn the feature space.
You may be thinking: "How is this different from the example in
Chapter 3 where we took the square of the feature values?"
There is a difference: Neurons can only perform affine transformations in the form
w T x + b. Therefore, an operation like x 2 , although simple, is still impossible.
This means we need a different transformation to be able to effectively increase
dimensionality and, more importantly, to twist and turn the feature space. That’s
the role of the activation function, as you’ve probably already guessed.
332 | Bonus Chapter: Feature Space