03: Linear Algebra - Review

Previous Next Index

Matrices - overview
  • Matrix elements
    • A(i,j) = entry in ith row and jth column

  • Provides a way to organize, index and access a lot of data
Vectors - overview

Matrix manipulation

  • Division by a scalar
    • Same as multiplying a matrix by 1/4
    • Each element is divided by the scalar
  • Combination of operands
    • Evaluate multiplications first
  • Matrix by vector multiplication
    • [3 x 2] matrix * [2 x 1] vector
      • New matrix is [3 x 1]
        • More generally if [a x b] * [b x c]
          • Then new matrix is [a x c]
      • How do you do it?
        • Take the two vector numbers and multiply them with the first row of the matrix
          • Then add results together - this number is the first number in the new vector
        • The multiply second row by vector and add the results together
        • Then multiply final row by vector and add them together
  • Detailed explanation
    • A * x = y
      • A is m x n matrix
      • x is n x 1 matrix
      • n must match between vector and matrix
        • i.e. inner dimensions must match
      • Result is an m-dimensional vector
    • To get yi - multiply A's ith row with all the elements of vector x and add them up
  • Neat trick
    • Say we have a data set with four values
    • Say we also have a hypothesis hθ(x) = -40 + 0.25x
      • Create your data as a matrix which can be multiplied by a vector
      • Have the parameters in a vector which your matrix can be multiplied by
    • Means we can do 
      • Prediction = Data Matrix * Parameters
      • Here we add an extra column to the data with 1s - this means our θ0 values can be calculated and expressed
  • The diagram above shows how this works
    • This can be far more efficient computationally than lots of for loops
    • This is also easier and cleaner to code (assuming you have appropriate libraries to do matrix multiplication)
  • Matrix-matrix multiplication
    • General idea
      • Step through the second matrix one column at a time
      • Multiply each column vector from second matrix by the entire first matrix, each time generating a vector
      • The final product is these vectors combined (not added or summed, but literally just put together)
    • Details
      • A x B = C
        • A = [m x n]
        • B = [n x o]
        • C = [m x o]
          • With vector multiplications o = 1
      • Can only multiply matrix where columns in A match rows in B
    • Mechanism
      • Take column 1 of B, treat as a vector
      • Multiply A by that column - generates an [m x 1] vector
      • Repeat for each column in B
        • There are o columns in B, so we get o columns in C
    • Summary
      • The i th column of matrix C is obtained by multiplying A with the i th column of B
    • Start with an example
    • A x B
  • Initially
    • Take matrix A and multiply by the first column vector from B
    • Take the matrix A and multiply by the second column vector from B

  • 2 x 3 times 3 x 2 gives you a 2 x 2 matrix
Implementation/use
  • House prices, but now we have three hypothesis and the same data set
  • To apply all three hypothesis to all data we can do this efficiently using matrix-matrix multiplication
    • Have
      • Data matrix
      • Parameter matrix
    • Example
      • Four houses, where we want to predict the prize
      • Three competing hypotheses
      • Because our hypothesis are one variable, to make the matrices match up we make our data (houses sizes) vector into a 4x2 matrix by adding an extra column of 1s

  • What does this mean
    • Can quickly apply three hypotheses at once, making 12 predictions
    • Lots of good linear algebra libraries to do this kind of thing very efficiently
Matrix multiplication properties
  • Can pack a lot into one operation
    • However, should be careful of how you use those operations
    • Some interesting properties
  • Commutativity
    • When working with raw numbers/scalars multiplication is commutative
      • 3 * 5 == 5 * 3
    • This is not true for matrix
      • A x B != B x A
      • Matrix multiplication is not commutative
  • Associativity
    • 3 x 5 x 2 == 3 x 10 = 15 x 2
      • Associative property
    • Matrix multiplications is associative
      • A x (B x C) == (A x B) x C
  • Identity matrix
    • 1 is the identity for any scalar
      • i.e. 1 x z = z 
        • for any real number
    • In matrices we have an identity matrix called I
      • Sometimes called I{n x n}
  • See some identity matrices above
    • Different identity matrix for each set of dimensions
    • Has
      • 1s along the diagonals
      • 0s everywhere else
    • 1x1 matrix is just "1"
Inverse and transpose operations
  • Matrix transpose
    • Have matrix A (which is [n x m]) how do you change it to become [m x n] while keeping the same values
      • i.e. swap rows and columns!
    • How you do it;
      • Take first row of A - becomes 1st column of AT
      • Second row of A - becomes 2nd column...
    • A is an m x n matrix
      • B is a transpose of A
      • Then B is an n x m matrix
      • A(i,j) = B(j,i)