08: Neural Networks - Representation

Neural networks - Overview and summary

Why do we need neural networks?

Say we have a complex supervised learning classification problem
- Can use logistic regression with many polynomial terms
- Works well when you have 1-2 features
- If you have 100 features

e.g. our housing example
- 100 house features, predict odds of a house being sold in the next 6 months
- Here, if you included all the quadratic terms (second order)
  - There are lots of them (x₁² ,x₁x₂, x₁x₄ ..., x₁x₁₀₀)
  - For the case of n = 100, you have about 5000 features
  - Number of features grows O(n²)
  - This would be computationally expensive to work with as a feature set
A way around this to only include a subset of features
- However, if you don't have enough features, often a model won't let you fit a complex dataset
If you include the cubic terms
- e.g. (x₁²x₂, x₁x₂x₃, x₁x₄x₂₃ etc)
- There are even more features grows O(n³)
- About 170 000 features for n = 100
Not a good way to build classifiers when n is large

Example: Problems where n is large - computer vision

Computer vision sees a matrix of pixel intensity values
- Look at matrix - explain what those numbers represent
To build a car detector
- Build a training set of
  - Not cars
  - Cars
- Then test against a car
How can we do this
- Plot two pixels (two pixel locations)
- Plot car or not car on the graph

Need a non-linear hypothesis to separate the classes
Feature space
- If we used 50 x 50 pixels --> 2500 pixels, so n = 2500
- If RGB then 7500
- If 100 x 100 then --> 50 000 000 (quadratic) features
Too big - wayyy too big
- So - simple logistic regression here is not appropriate for large complex systems
- Neural networks are much better for a complex nonlinear hypothesis even when feature space is huge

Neurons and the brain

Neural networks (NNs) were originally motivated by looking at machines which replicate the brain's functionality
- Looked at here as a machine learning technique
Origins
- To build learning systems, why not mimic the brain?
- Used a lot in the 80s and 90s
- Popularity diminished in late 90s
- Recent major resurgence
  - NNs are computationally expensive, so only recently large scale neural networks became computationally feasible
Brain
- Does loads of crazy things
  - Hypothesis is that the brain has a single learning algorithm
- Evidence for hypothesis
  - Auditory cortex --> takes sound signals
    - If you cut the wiring from the ear to the auditory cortex
    - Re-route optic nerve to the auditory cortex
    - Auditory cortex learns to see
  - Somatosensory cortex (touch processing)
    - If you rewire optic nerve to somatosensory cortex then it learns to see
- With different tissue learning to see, maybe they all learn in the same way
  - Brain learns by itself how to learn
Other examples
- Seeing with your tongue
  - Brainport
    - Grayscale camera on head
    - Run wire to array of electrodes on tongue
    - Pulses onto tongue represent image signal
    - Lets people see with their tongue
- Human echolocation
  - Blind people being trained in schools to interpret sound and echo
  - Lets them move around
- Haptic belt direction sense
  - Belt which buzzes towards north
  - Gives you a sense of direction
Brain can process and learn from data from any source

Model representation 1

How do we represent neural networks (NNs)?
- Neural networks were developed as a way to simulate networks of neurones
What does a neurone look like

Three things to notice
- Cell body
- Number of input wires (dendrites)
- Output wire (axon)
Simple level
- Neurone gets one or more inputs through dendrites
- Does processing
- Sends output down axon
Neurons communicate through electric spikes
- Pulse of electricity via axon to another neurone

Artificial neural network - representation of a neurone

In an artificial neural network, a neurone is a logistic unit
- Feed input via input wires
- Logistic unit does computation
- Sends output down output wires
That logistic computation is just like our previous logistic regression hypothesis calculation

Very simple model of a neuron's computation
- Often good to include an x₀ input - the bias unit
  - This is equal to 1
This is an artificial neurone with a sigmoid (logistic) activation function
- Θ vector may also be called the weights of a model
The above diagram is a single neurone
- Below we have a group of neurones strung together

Here, input is x₁, x₂and x₃
- We could also call input activation on the first layer - i.e. (a₁¹, a₂¹and a₃¹)
- Three neurones in layer 2 (a₁², a₂²and a₃²)
- Final fourth neurone which produces the output
  - Which again we *could* call a₁³
First layer is the input layer
Final layer is the output layer - produces value computed by a hypothesis
Middle layer(s) are called the hidden layers
- You don't observe the values processed in the hidden layer
- Not a great name
- Can have many hidden layers

Neural networks - notation

a_i^(j) - activation of unit i in layer j
- So, a₁² - is the activation of the 1st unit in the second layer
- By activation, we mean the value which is computed and output by that node
Θ^(j) - matrix of parameters controlling the function mapping from layer j to layer j + 1
- Parameters for controlling mapping from one layer to the next
- If network has
  - s_j units in layer j and
  - s_j+1 units in layer j + 1
  - Then Θ^j will be of dimensions [s_{j+1 X}s_j + 1]
    - Because
      - s_j+1is equal to the number of units in layer (j + 1)
      - is equal to the number of units in layer j, plus an additional unit
- Looking at the Θ matrix
  - Column length is the number of units in the following layer
  - Row length is the number of units in the current layer + 1 (because we have to map the bias unit)
  - So, if we had two layers - 101 and 21 units in each
    - Then Θ^j would be = [21 x 102]
What are the computations which occur?
- We have to calculate the activation for each node
- That activation depends on
  - The input(s) to the node
  - The parameter associated with that node (from the Θ vector associated with that layer)
Below we have an example of a network, with the associated calculations for the four nodes below

a_{1}^{(2)} = g (Θ_{10}^{(1)} x_{0} + Θ_{11}^{(1)} x_{1} + Θ_{12}^{(1)} x_{2} + Θ_{13}^{(1)} x_{3})

a_{2}^{(2)} = g (Θ_{20}^{(1)} x_{0} + Θ_{21}^{(1)} x_{1} + Θ_{22}^{(1)} x_{2} + Θ_{23}^{(1)} x_{3})

a_{3}^{(2)} = g (Θ_{30}^{(1)} x_{0} + Θ_{31}^{(1)} x_{1} + Θ_{32}^{(1)} x_{2} + Θ_{33}^{(1)} x_{3})

h_{Θ} (x) = a_{1}^{(3)} = g (Θ_{10}^{(2)} a_{0}^{(2)} + Θ_{11}^{(2)} a_{1}^{(2)} + Θ_{12}^{(2)} a_{2}^{(2)} + Θ_{13}^{(2)} a_{3}^{(2)})

The three layer-2 activations, then the hypothesis computed from them. Each is a logistic unit over the previous layer plus its bias term.

As you can see
- We calculate each of the layer-2 activations based on the input values with the bias term (which is equal to 1)
  - i.e. x₀ to x₃
- We then calculate the final hypothesis (i.e. the single node in layer 3) using exactly the same logic, except in input is not x values, but the activation values from the preceding layer
The activation value on each hidden unit (e.g. a₁² ) is equal to the sigmoid function applied to the linear combination of inputs
- Three input units
  - So Θ⁽¹⁾is the matrix of parameters governing the mapping of the input units to hidden units
    - Θ⁽¹⁾ here is a [3 x 4] dimensional matrix
- Three hidden units
  - Then Θ⁽²⁾is the matrix of parameters governing the mapping of the hidden layer to the output layer
    - Θ⁽²⁾ here is a [1 x 4] dimensional matrix (i.e. a row vector)
- One output unit
Something conceptually important (that I hadn't really grasped the first time) is that
- Every input/activation goes to every node in following layer
  - Which means each "layer transition" uses a matrix of parameters with the following significance
    - For the sake of consistency with later nomenclature, we're using j,i and l as our variables here (although later in this section we use j to show the layer we're on)
    - Θ_ji^l
      - j (first of two subscript numbers)= ranges from 1 to the number of units in layer l+1
      - i (second of two subscript numbers) = ranges from 0 to the number of units in layer l
      - l is the layer you're moving FROM
    - This is perhaps more clearly shown in my slightly over the top example below
For example
- Θ₁₃¹= means
  - 1 - we're mapping to node 1 in layer l+1
  - 3 - we're mapping from node 3 in layer l
  - 1 - we're mapping from layer 1

Model representation II

Here we'll look at how to carry out the computation efficiently through a vectorized implementation. We'll also consider

why NNs are good and how we can use them to learn complex non-linear things

Below is our original problem from before
- Sequence of steps to compute output of hypothesis are the equations below

a_{1}^{(2)} = g (Θ_{10}^{(1)} x_{0} + Θ_{11}^{(1)} x_{1} + Θ_{12}^{(1)} x_{2} + Θ_{13}^{(1)} x_{3})

a_{2}^{(2)} = g (Θ_{20}^{(1)} x_{0} + Θ_{21}^{(1)} x_{1} + Θ_{22}^{(1)} x_{2} + Θ_{23}^{(1)} x_{3})

a_{3}^{(2)} = g (Θ_{30}^{(1)} x_{0} + Θ_{31}^{(1)} x_{1} + Θ_{32}^{(1)} x_{2} + Θ_{33}^{(1)} x_{3})

h_{Θ} (x) = a_{1}^{(3)} = g (Θ_{10}^{(2)} a_{0}^{(2)} + Θ_{11}^{(2)} a_{1}^{(2)} + Θ_{12}^{(2)} a_{2}^{(2)} + Θ_{13}^{(2)} a_{3}^{(2)})

The three layer-2 activations, then the hypothesis computed from them. Each is a logistic unit over the previous layer plus its bias term.

Define some additional terms
- z₁² = Θ₁₀¹x₀+ Θ₁₁¹x₁+ Θ₁₂¹x₂+ Θ₁₃¹x3
- Which means that
  - a₁² = g(z₁²)
- NB, superscript numbers are the layer associated
Similarly, we define the others as
- z₂² and z₃²
- These values are just a linear combination of the values
If we look at the block we just redefined
- We can vectorize the neural network computation
- So lets define
  - x as the feature vector x
  - z² as the vector of z values from the second layer

x = [\begin{matrix} x_{0} \\ x_{1} \\ x_{2} \\ x_{3} \end{matrix}] z^{(2)} = [\begin{matrix} z_{1}^{(2)} \\ z_{2}^{(2)} \\ z_{3}^{(2)} \end{matrix}]

x = np.array([1, 0.5, -1.2, 0.8])   # x0 = 1 plus three features
Theta1 = np.array([[ 0.1, -0.4,  0.6,  0.2],   # [3 x 4]: three hidden units,
                   [-0.3,  0.5, -0.1,  0.7],   # each seeing all four inputs
                   [ 0.2,  0.1,  0.3, -0.5]])

z2 = Theta1 @ x                       # [3]  - one z per hidden unit
a2 = g(z2)                            # [3]  - their activations
a2 = np.concatenate(([1], a2))        # [4]  - add the bias unit a0 = 1

The two vectorized steps above, as code: one matrix multiply computes every hidden unit's z at once, one call to g turns them into activations.

z² is a 3x1 vector
We can vectorize the computation of the neural network as as follows in two steps
- z² = Θ⁽¹⁾x
  - i.e. Θ⁽¹⁾ is the matrix defined above
  - x is the feature vector
- a² = g(z⁽²⁾)
  - To be clear, z² is a 3x1 vector
  - a² is also a 3x1 vector
  - g() applies the sigmoid (logistic) function element wise to each member of the z² vector
To make the notation with input layer make sense;
- a¹ = x
  - a¹ is the activations in the input layer
  - Obviously the "activation" for the input layer is just the input!
- So we define x as a¹ for clarity
  - So
    - a¹ is the vector of inputs
    - a² is the vector of values calculated by the g(z²) function
Having calculated the z² vector, we need to calculate a₀² for the final hypothesis calculation

h_{Θ} (x) = g (Θ_{10}^{(2)} a_{0}^{(2)} + Θ_{11}^{(2)} a_{1}^{(2)} + Θ_{12}^{(2)} a_{2}^{(2)} + Θ_{13}^{(2)} a_{3}^{(2)})

a₀⁽²⁾ is the bias unit, added to a⁽²⁾ to make it a 4x1 vector — it is picked out in red on the original slide.

To take care of the extra bias unit add a₀² = 1
- So add a₀² to a² making it a 4x1 vector
So,
- z³ = Θ²a²
  - This is the inner term of the above equation
- h_Θ(x) = a³ = g(z³)
This process is also called forward propagation
- Start off with activations of input unit
  - i.e. the x vector as input
- Forward propagate and calculate the activation of each layer sequentially
- This is a vectorized version of this implementation

Neural networks learning its own features

Diagram below looks a lot like logistic regression

Layer 3 is a logistic regression node
- The hypothesis output = g(Θ₁₀²a₀² + Θ₁₁²a₁² + Θ₁₂²a₂² + Θ₁₃²a₃²)
- This is just logistic regression
  - The only difference is, instead of input a feature vector, the features are just values calculated by the hidden layer
The features a₁², a₂², and a₃² are calculated/learned - not original features
So the mapping from layer 1 to layer 2 (i.e. the calculations which generate the a² features) is determined by another set of parameters - Θ¹
- So instead of being constrained by the original input features, a neural network can learn its own features to feed into logistic regression
- Depending on the Θ¹ parameters you can learn some interesting things
  - Flexibility to learn whatever features it wants to feed into the final logistic regression calculation
    - So, if we compare this to previous logistic regression, you would have to calculate your own exciting features to define the best way to classify or describe something
    - Here, we're letting the hidden layers do that, so we feed the hidden layers our input values, and let them learn whatever gives the best final result to feed into the final output layer
As well as the networks already seen, other architectures (topology) are possible
- More/less nodes per layer
- More layers
- Once again, layer 2 has three hidden units, layer 3 has 2 hidden units by the time you get to the output layer you get very interesting non-linear hypothesis

Some of the intuitions here are complicated and hard to understand
- In the following lectures we're going to go through a detailed example to understand how to do non-linear analysis

Neural network example - computing a complex, nonlinear function of the input

Non-linear classification: XOR/XNOR
- x₁, x₂ are binary

Example on the right shows a simplified version of the more complex problem we're dealing with (on the left)
We want to learn a non-linear decision boundary to separate the positive and negative examples

y = x₁ XOR x₂

x₁ XNOR x₂

Where XNOR = NOT (x₁ XOR x₂)

Positive examples when both are true and both are false
- Let's start with something a little more straight forward...
- Don't worry about how we're determining the weights (Θ values) for now - just get a flavor of how NNs work

Neural Network example 1: AND function

Simple first example

Can we get a one-unit neural network to compute this logical AND function? (probably...)
- Add a bias unit
- Add some weights for the networks
  - What are weights?
    - Weights are the parameter values which multiply into the input nodes (i.e. Θ)

Sometimes it's convenient to add the weights into the diagram
- These values are in fact just the Θ parameters so
  - Θ₁₀¹ = -30
  - Θ₁₁¹ = 20
  - Θ₁₂¹ = 20
- To use our original notation
Look at the four input values

h_Θ(x) = g(−30 + 20x₁ + 20x₂)
x₁	x₂	h_Θ(x)
0	0	g(−30) ≈ 0
0	1	g(−10) ≈ 0
1	0	g(−10) ≈ 0
1	1	g(10) ≈ 1

Only (1,1) gives a positive output — so h_Θ(x) ≈ x₁ AND x₂.

import numpy as np

def g(z):   # the sigmoid, as everywhere else
    return 1 / (1 + np.exp(-z))

Theta = np.array([-30, 20, 20])   # the weights drawn on the network above

for x1, x2 in [(0, 0), (0, 1), (1, 0), (1, 1)]:
    h = g(Theta @ np.array([1, x1, x2]))
    # h -> 0.000, 0.000, 0.000, 1.000 : only (1,1) fires

The AND table above, computed. The weights push the sigmoid deep into saturation, so the outputs are 0 and 1 to three decimal places.

So, as we can see, when we evaluate each of the four possible input, only (1,1) gives a positive output

Neural Network example 2: NOT function

How about negation?

Negation is achieved by putting a large negative weight in front of the variable you want to negate

Neural Network example 3: XNOR function

So how do we make the XNOR function work?
- XNOR is short for NOT XOR
  - i.e. NOT an exclusive or, so either go big (1,1) or go home (0,0)
- So we want to structure this so the inputs which produce a positive output are
  - AND (i.e. both true)
    OR
  - Neither (which we can shortcut by saying not only one being true)
So we combine these into a neural network as shown below;

Combining the three gates: a₁⁽²⁾ = x₁ AND x₂, a₂⁽²⁾ = (NOT x₁) AND (NOT x₂), h_Θ(x) = a₁⁽²⁾ OR a₂⁽²⁾
x₁	x₂	a₁⁽²⁾	a₂⁽²⁾	h_Θ(x)
0	0	0	1	1
0	1	0	0	0
1	0	0	0	0
1	1	1	0	1

The output is 1 exactly when the inputs agree — XNOR — which a single logistic unit cannot express. Gate weights: AND (−30, 20, 20), NOR (10, −20, −20), OR (−10, 20, 20).

Theta1 = np.array([[-30,  20,  20],   # row 1: AND
                   [ 10, -20, -20]])  # row 2: (NOT x1) AND (NOT x2)
Theta2 = np.array([-10, 20, 20])      # output layer: OR

def forward(x1, x2):
    a1 = np.array([1, x1, x2])              # input plus bias
    a2 = g(Theta1 @ a1)                     # both gates at once
    return g(Theta2 @ np.concatenate(([1], a2)))

[round(forward(x1, x2)) for x1, x2 in [(0, 0), (0, 1), (1, 0), (1, 1)]]
# [1, 0, 0, 1]  - XNOR, from two layers of logistic units

The whole XNOR network, forward-propagated. Layer 1 computes the two gates, layer 2 ORs them together — a non-linear function no single logistic unit can express.

Simplez!

Neural network intuition - handwritten digit classification

Yann LeCun = machine learning pioneer
Early machine learning system was postcode reading
- Hilarious music, impressive demonstration!

Multiclass classification

Multiclass classification is, unsurprisingly, when you distinguish between more than two categories (i.e. more than 1 or 0)
With handwritten digit recognition problem - 10 possible categories (0-9)
- How do you do that?
- Done using an extension of one vs. all classification
Recognizing pedestrian, car, motorbike or truck
- Build a neural network with four output units
- Output a vector of four numbers
  - 1 is 0/1 pedestrian
  - 2 is 0/1 car
  - 3 is 0/1 motorcycle
  - 4 is 0/1 truck
- When image is a pedestrian get [1,0,0,0] and so on
Just like one vs. all described earlier
- Here we have four logistic regression classifiers

Training set here is images of our four classifications
- While previously we'd written y as an integer {1,2,3,4}
- Now represent y as
$y^{(i)} is one of [\begin{matrix} 1 \\ 0 \\ 0 \\ 0 \end{matrix}], [\begin{matrix} 0 \\ 1 \\ 0 \\ 0 \end{matrix}], [\begin{matrix} 0 \\ 0 \\ 1 \\ 0 \end{matrix}], [\begin{matrix} 0 \\ 0 \\ 0 \\ 1 \end{matrix}]$ One column per class, rather than an integer in {1, 2, 3, 4}.