Problem Set 3: Matrix Calculus
                                                                           M.Phil. Premath, September 2007
               Matrices are typically used to represent big systems of linear equations. Of’course one can
               use calculus in such systems. Let f (X) be a scalar function of the (n × m) matrix X.Then
               ∂f(X)/∂X is an (n×m) matrix whose (i,j)-th element is ∂f(X)/∂xij.
                                                                                     0
               Exercise 1 Let x,a denote two (n×1) vectors. Denote f(x)=ax. Show that
                                                          ∂f(x) =a
                                                            ∂x
               Exercise 2 If A denotes (n×m) matrix and x is (m×1) vector, verify that
                                                         ∂(Ax) =A
                                                              0
                                                           ∂x
                                        ∙       ¸
               Exercise 3 Take A =        21.Verify
                                          31
                                                       0
                                                    ∂xAx               0
                                                      ∂x    =(A+A)x
                          ∙ x ¸
               where x =      1   is an (2 × 1) vector.
                             x
                              2
               Exercise 4 Take matrix                       ⎡         ⎤
                                                                23
                                                            ⎣         ⎦
                                                       X= 45
                                                               −10
                            0
               Compute XX.
                                          0
               Exercise 5 Show that XX is symmetric. Compute
                                                            0   0
                                                         ∂b (XX)b
                                                              ∂b
               using formula from excercise 3.
               Exercise 6 Suppose that for given (n×1) vector y and (n×k) full column rank matrix X
               we want to find (k ×1) vector of coefficients b, such that the linear combination of column
                                                                                                           2
               vectors of X (i.e. Xb)isclosesttovectory, i.e. we want to minimise ky−Xbk ≡
                                                0
               hy−Xb,y−Xbi=(y−Xb) (y−Xb).Wecanproceed in two ways:
                                                      2
                                              ∂ky−Xbk             b
                 (a) Use calculus: Compute         0    and find b for which this derivation is zero.
                                                 ∂b
                                                               1
                (b) Use so called orthogonal projection theorem, which says that:
                    b                      2       ³       b´                                                     k
                    b=argminky−Xbk ⇐⇒ y−Xb isorthogonal to any linear combination Xb, b∈R
                                k
                             b∈R
                                                b
                    I.e. we have to find vector b,whichsatisfies following orthogonality conditions
                                              D           E   ³         ´
                                                     b                b 0
                                               y−Xb,X ≡ y−Xb X=0
                                                                                k
                                                            2