ECE595 / STAT598: Machine Learning I
                          Lecture 01: Linear Regression
                                            Spring 2020
                                           Stanley Chan
                            School of Electrical and Computer Engineering
                                         Purdue University
                                                                              c
                                                                             
Stanley Chan 2020. All Rights Reserved.
                                                                                              1/22
   Outline
                                                                             c
                                                                            
Stanley Chan 2020. All Rights Reserved.
                                                                                             2/22
   Outline
   Mathematical Background
         Lecture 1: Linear regression: A basic data analytic tool
         Lecture 2: Regularization: Constraining the solution
         Lecture 3: Kernel Method: Enabling nonlinearity
   Lecture 1: Linear Regression
         Linear Regression
               Notation
               Loss Function
               Solving the Regression Problem
         Geometry
               Projection
               Minimum-Norm Solution
               Pseudo-Inverse
                                                                             c
                                                                            
Stanley Chan 2020. All Rights Reserved.
                                                                                             3/22
   Basic Notation
           Scalar: a,b,c ∈ R
           Vector: ❛,❜,❝ ∈ Rd
           Matrix: ❆,❇,❈ ∈ RN×d; Entries are a or [❆] .
                                                                      ij          ij
           Rows and Columns
                           |        |             |                         — (①1)T                —
                                                                              — (①2)T                —
                   ❆=❛ ❛ ... ❛ , and ❆=                                                                .
                               1     2              d                                      .             
                                                                                           .             
                              |      |             |                                        .
                                                                                           N T
                                                                                 — (① )               —
           {❛j}: The j-th feature. {①n}: The n-th sample.
           Identity matrix ■
           All-one vector 1 and all-zero vector 0
           Standard basis ❡i.
                                                                                              c
                                                                                             
Stanley Chan 2020. All Rights Reserved.
                                                                                                                 4/22