Aspects of Music
                              Lecture                                                   Piece of music
                          Music Processing
                      Chord Recognition
                                                                                                            Rhythm
                           Meinard Müller                                     Melody
                    International Audio Laboratories Erlangen
                    meinard.mueller@audiolabs-erlangen.de
                                                                                             Harmony
         Harmony: The Basis of Music                                     Musical Chords
           Pachelbel’s Canon
                                                                          Combination of three or more tones which sound 
                                                                           simultaneously
                                                                          Chord classes 
                                                                           – Triads including major, minor, diminished, augmented chords
                                                                           – Many other more complex chords such as seventh chords
                                                                          Here: focus on major and minor triads
           Coversong Die Eine (Die Firma)
         Musical Chords                                                  Musical Chords
          The C major chord                                              The C minor chord
          Derived from the C major scale                                 Derived from the C minor scale
           C ---- the root                                                C ---- the root
           E ---- the (major)third                                        Eb---- the (minor)third
           G  ---- the fifth                                              G  ---- the fifth
               Musical Chords                                                                                         Chord Recognition
               Structure of the 24 major/minor chords
                                                                                                                       Development of automatic methods for the 
                                                                                                                         harmonic analysis of audio data
                                                                                                                       Applications in the field of music information 
                      01234567891011                                                                                     retrieval:
                      CC#DD#EFF#GG#AA#B                                                                                   – music segmentation
             C major                                                                                                      – cover song identification
                                                                                                                          – audio matching
             C minor                                                                                                      – music structure analysis 
                                                                                                                          – …
               Chord Recognition                                                                                      Chord Recognition
                   Signal                                                 Result                                          Signal            Feature        Classification        Result
                                                                                                                                           Extraction
                                                       Segmentation                                                                    Chroma features       Pattern matching
                                              Chord label assignment   C    F G C                                                       Type                 Template
                                                                                                                                        Resolution           Gaussian
                                                                                                                                        Compression          Hidden Markov Models
                                                                                                                                        Smoothing            Graphical Models 
               Chord Recognition                                                                                      Baseline Method for Chord Recognition
               Given:    Audio file                                                                                   Chord templates 24 major/minor chords
               Output:  Segmentation and chord labeling                                                                            C      C#      D      D#     ……      C      C#     ……
                                                                                                                                 major   major  major   major          minor  minor
                                                                                                                         B         0      0       0      0      ……      0       0     ……
                                                                                                                         A#        0      0       0      1      ……      0       0     ……
                                                                                                                         A         0      0       1      0      ……      0       0     ……
                                                                                                                         G#        0      1       0      0      ……      0       1     ……
                                                                                                                         G         1      0       0      1      ……      1       0     ……
                                                                                                                         F#        0      0       1      0      ……      0       0     ……
                          C      A:min E:min FC FCA:min E:min
                              G                       G      G                                                           F         0      1       0      0      ……      0       0     ……
                                                                                                                         E         1      0       0      0      ……      0       1     ……
                                                                                                                         D#        0      0       0      1      ……      1       0     ……
                                                                                                                         D         0      0       1      0      ……      0       0     ……
                                       CGA:minE:min                 G                                                    C#        0      1       0      0      ……      0       1     ……
                      F  C    FG                         F  CF
                                                                                                                         C         1      0       0      0      ……      1       0     ……
                             Baseline Method for Chord Recognition                                                                                                                                                                        Baseline Method for Chord Recognition
                              Chord templates 24 major/minor chords                                                                                                                                                                                 24 chord templates
                                                                                                                                                                                                                                                                                                                                             Chromafeature 
                                                                                                                                                                                                                                                  (12 major, 12 minor)                                                               extraction (framewise)
                                  Chroma
                                                                                            Chord labels
                             Baseline Method for Chord Recognition                                                                                                                                                                        Baseline Method for Chord Recognition
                                       24 chord templates                                                                       Chromafeature                                                                                                       24 chord templates                                                                       Chromafeature 
                                      (12 major, 12 minor)                                                              extraction (framewise)                                                                                                    (12 major, 12 minor)                                                               extraction (framewise)
                                                         Compute for each frame the distance of                                                                                                                                                                       Compute for each frame the distance of 
                                                         the feature vector to the 24 templates                                                                                                                                                                       the feature vector to the 24 templates
                                                                                                                                                                                                                                                                   Selected chord according to template with 
                                                                                                                                                                                                                                                                minimal distance to respective feature vector
                             Problems in Chord Recognition                                                                                                                                                                                Problems in Chord Recognition
                             Problem: Transitions between subsequent chord                                                                                                                                                               Problem: Monphonic musical passages
                             Example: Chopin Mazurka Op. 68 No.3                                                                                                                                                                         Example: Excerpt of Wagner’s Meistersinger
                                                                                                                                                                                                                                          Chromagram
                                                                                                                                                         Correct                                                                                                                                                                        1
                                                                                                                                                                                                                                           B 
                                                                                                                                                         False positive                                                                   A#                                                                                            0.9
                                                                                                                                                                                                                                           A                                                                                            0.8
                                                                                                                                                         False negative                                                                   G#                                                                                            0.7
                                                                                                                                                                                                                                           G                                                                                            0.6
                                                                                                                                                                                                                                           F#
                                                                                                                                                                                                                                                                                                                                        0.5
                                                                                                                                                                                                                                           F 
                                                                                                                                                                                                                                           E                                                                                            0.4
                                                                                                                                                                                                                                          D#                                                                                            0.3
                                                                                                                                                                                                                                           D                                                                                            0.2
                                                                                                                                                                                                                                          C#                                                                                            0.1
                                                                                                                                                                                                                                           C 
                                                                                                                                                                                                                                                                                                                                        0
                                                                                                                                                                                                                                             0            10           20            30           40            50           60
                             Problems in Chord Recognition                                                                                                                                                                                  Problems in Chord Recognition
                             Problem: Frame-wise chord analysis may not be meaningful                                                                                                                                                       Problem: Ambiguity of chords
                             Example: Bach: Prelude C major, BWV 846
                                                                                                                                                                                                                                                                A minor                                          C major                                           E minor
                                                                                                                                                                                                                                                                        A
                                                                                                                                                                                                                                                                         C                                               C  
                                                                                                                                                                                                                                                                         E                                               E                                                 E
                              Problem: Broken chords                                                                                                                                                                                                                                                                     G                                                 G
                                                                                                                                                                                                                                                                                                                                                                           B
                                   →  Measure-wise chord analysis necessary
                             Problems in Chord Recognition                                                                                                                                                                                  Problems in Chord Recognition
                             Problem: Reduction to the 24 major/minor chords makes                                                                                                                                                          Problem: Tuning problems
                             the recognition of more complex chords difficult/impossible!                                                                                                                                                   Example: Excerpt of Wagner’s Meistersinger
                             Example: Prelude C major, BWV 846, mm.19-25
                                                                                                                                                                                                                                            Chromagram (from MIDI)
                                                                                                                                                                                                                                                                                                                                           1
                                                                                                                                                                                                                                              B 
                                                                                                                                                                                                                                                                                                                                           0.9
                                                                                                                                                                                                                                              A#
                                                                                                                                                                                                                                              A                                                                                            0.8
                                                                                                                                                                                                                                              G#                                                                                           0.7
                                                                                                                                                                                                                                              G                                                                                            0.6
                                                                                                                                                                                                                                              F#
                                                                                                                                                                                                                                                                                                                                           0.5
                                                                                                                                                                                                                                              F 
                                                                                                                                                                                                                                                                                                                                           0.4
                                                                                                                                                                                                                                              E 
                                                                                                                                                                                                                                                                                                                                           0.3
                                                                                                                                                                                                                                              D#
                                                                                                                                                                                                                                              D                                                                                            0.2
                                                                                                                                                                                                                                              C#                                                                                           0.1
                                                                                                                                                                                                                                              C                                                                                            0
                                                                                                                                                                                                                                                0            10            20           30            40           50            60
                             Problems in Chord Recognition                                                                                                                                                                                  Problems in Chord Recognition
                             Problem: Tuning problems                                                                                                                                                                                       Problem: Tuning problems
                             Example: Excerpt of Wagner’s Meistersinger                                                                                                                                                                     Example: Excerpt of Wagner’s Meistersinger
                             Chromagram (from MIDI)                                                                                  Problem:                                                                                               Chromagram (from MIDI)                                                                                  Problem: 
                                                                                                                            1        Audio is tuned more than                                                                                                                                                                              1        Audio is tuned more than 
                                B                                                                                                                                                                                                             B 
                               A#                                                                                           0.9      half a semi-tone upwards                                                                                                                                                                              0.9      half a semi-tone upwards
                                                                                                                                                                                                                                              A#
                                A                                                                                           0.8                                                                                                               A                                                                                            0.8
                               G#                                                                                           0.7                                                                                                               G#                                                                                           0.7      Solution:
                               G                                                                                            0.6                                                                                                               G 
                                                                                                                                                                                                                                                                                                                                           0.6
                               F#                                                                                                                                                                                                             F#                                                                                                    Adjust frequency binning
                                                                                                                            0.5                                                                                                                                                                                                            0.5
                                F                                                                                                                                                                                                              F 
                                E                                                                                           0.4                                                                                                                                                                                                            0.4      when computing pitch
                                                                                                                                                                                                                                              E 
                               D#                                                                                           0.3                                                                                                               D#                                                                                           0.3      features.
                                D                                                                                           0.2                                                                                                               D                                                                                            0.2
                               C#                                                                                           0.1                                                                                                               C#
                                                                                                                                                                                                                                                                                                                                           0.1
                                C                                                                                                                                                                                                             C 
                                                                                                                            0                                                                                                                                                                                                              0
                                  0           10            20           30           40           50            60                                                                                                                              0           10            20           30            40           50            60