On Projections of Gaussian Distributions using Maximum Likelihood Criteria

Transcription

1 On Projetion of Gauian Ditribution uing Maximum Likelihood Criteria Haolang Zhou, Damiano Karako, Sanjeev Khudanpur, Andrea G. Andreou and Carey E. Priebe Department of Eletrial and Computer Engineering and Center for Language and Speeh Proeing Department of Applied Mathemati and Statiti John Hopkin Univerity, Baltimore, MD, USA Abtrat Generative tatitial model with a very large number of parameter are frequently ued in real-world data appliation, uh a large-voabulary peeh reognition LVCSR. Complex model are needed in order to apture the ubiquitou variability in the oberved ignal, but data parity aue ignifiant problem in their training. One way of dealing with data parity i to perform dimenionality redution of the oberved feature, with the goal of reduing the model parameter pae without arifiing performane. When the data are Gauian ditributed, the dimenionality redution an be done effiiently uing the maximum likelihood riterion; thi lead to the Heteroedati Linear Diriminant Analyi HLDA, whih i a natural extenion of Linear Diriminant Analyi LDA to the ae where the la-onditional Gauian have unequal ovariane matrie. A further extenion of HLDA to multiple tranform MLDA an alo be takled effiiently. Thi paper preent the theory behind HLDA and MLDA, and demontrate their performane with yntheti data. I. GEERATIVE MODELS I SPEECH RECOGITIO Speeh reognition i a omplex laifiation tak. The oberved ignal vetor A, whih repreent the aouti, i the reult of a aade of ignal proeing operation, and i parameterized in a way that preerve information about the word uttered, while being invariant to ertain kind of irrelevant variation e.g., mirophone. Modeling the obervation i uually done in a generative way, whih aume that A i the realization of a random proe, whoe parameter depend on the true label e.g., word identitie of the obervation. Inferene i uually done in a maximumapoteriori fahion, Ŵ = arg max P A WP W, W where P A W i the generative model of the aouti given the word equene W aouti model, and P W i the language model [1]. A number of reaonable independene aumption allow to fator P A W into a number of omponent, eah of whih orrepond to a onditional ditribution that model ome apet of the peeh prodution. The uually very large olletion of onditional ditribution i aumed to belong to a parameterized model family, and training aouti Thi work wa upported by ational Siene Foundation grant o CCF model amount to etimating the parameter of thee model. Mixture of Gauian ditribution with diagonal ovariane matrie are very frequently ued a the underlying model, beaue of their expreivene and effiieny in their training. See [1], [] for detail about how aouti model are trained. Thu, expreing the aouti vetor A a a equene of obervation {a i }, eah a i i generated given label w by an underlying proe M pa w = p w a = λ m φa; µ m w, Σ m w m=1 where M i the number of Gauian omponent in the mixture, and {µ 1,..., µ M }, {Σ 1,..., Σ M } are the mean and ovariane matrie of thee omponent. Thee an be etimated with the EM algorithm [3], with the objetive of maximizing the likelihood of the training data. Etimation of ovariane matrie uffer from high variane and i omputationally intenive when the dimenionality of the Gauian vetor i large e.g., of the order of thouand. For thi reaon, projetion into a pae of lower dimenionality i performed before training omplex aouti model with many mixture omponent. The high dimenionality arie in peeh reognition from onatenating together many feature, e.g., PLP/MFCC feature, PLP/MFCC feature from neighboring peeh frame, artiulatory feature, et. Sine many of thee feature are highly orrelated, the lower-dimenional projetion an be ued to deorrelate them a well, keeping only thoe feature whih arry information for dirimination between the lae and diarding the ret. Thi paper i mainly a preentation of the mathemati behind two popular method ued for projeting data: Linear Diriminant Analyi LDA [4], [5] and Heteroedati Linear Diriminant Analyi HLDA [6] a well a it variant, Multiple LDA MLDA [7]. HLDA ue maximization of likelihood of the non-projeted data a the riterion for etimating the tranform, and it ha been ued with ue in peeh reognition. In fat, a i hown in [8] and reviewed here, LDA an alo be derived a a maximum-likelihood olution, under a ontraint of equal ovariane matrie. A number of experiment with yntheti Gauian data demontrate the performane of the above heme, under a variety of ondition amount of training data, amount of

2 overlap of the lae. It i oberved that HLDA alway outperform LDA when the la-onditional ditribution have unequal ovariane matrie, and MLDA alway outperform HLDA. II. MATHEMATICAL PRELIMIARIES It i aumed that a training orpu exit, oniting of obervation olumn vetor x 1,..., x, eah belonging to IR n. There are C label lae {1,..., C}, and the label of obervation x j i denoted by j. The number of obervation of la i, and hene C =. The ample mean of la i denoted by µ, while the ample ovariane of la the within-la ovariane i denoted by Σ. Speifially, µ 1 x j, Σ 1 x j µ x j µ T, where a T i the tranpoe of matrix or vetor a. The global mean and variane are denoted by µ and Σ, repetively. The k-dimenional Gauian denity with mean µ and ovariane matrix Σ i denoted by φ k ; µ, Σ, or jut φ ; µ, Σ when the dimenionality i lear from the ontext. Projeting a vetor x into IR p i done by multiplying it with a p n matrix Θ p < n. Thu, y j = Θx j, j = 1,..., i the projetion of the j-th obervation. III. LIEAR PROJECTIOS FOR CLASSIFICATIO Two projetion method are epeially popular in peeh reognition: i Linear Diriminant Analyi LDA, and ii Heteroedati Linear Diriminant Analyi HLDA [6]. In addition, a natural extenion of HLDA i preented in [7] a iii Multiple LDA. Thee tehnique are reviewed in the ret of thi etion. A. Claial LDA For the -la problem, Linear Diriminant Analyi wa introdued by Fiher [4] and Rao [5] a a method for finding the mot diriminant projetion diretion whih maximize the ratio between the average between-la quared Eulidean ditane and the average within-la quared Eulidean ditane. That i, the goal i to etimate an 1 n matrix Θ 1 LDA, whih repreent the projetion diretion, uh that the ratio JΘ 1 LDA Θ 1 LDAµ µ Θ 1 LDAx j µ i maximized. More generally, the matrix Θ LDA that give the projetion to the p mot diriminant diretion an be determined by firt projeting the data to the p 1 mot diriminant diretion, and then finding the next mot diriminant diretion of the differene between the original and the projeted data. Then, the LDA objetive funtion beome JΘ = ΘBΘT ΘWΘ T, 1 where B µ µµ µ T and W Σ are the average between-la and within-la ovariane matrie, repetively. The olution of the maximization of 1 i a matrix Θ LDA whih i omputed by pot-multiplying W 1/ with a matrix, whoe row are the eigenvetor orreponding to the p larget eigenvalue of M = W 1/ BW 1/, provided that W i non-ingular. If it i ingular, a lowerdimenional ubpae an be identified uing Prinipal Component Analyi. B. Maximum-Likelihood Projetion of Gauian Data: Heteroedati Linear Diriminant Analyi Projetion of the data in IR p an be viewed a the proe of i firt tranforming the data into IR n, and ii keeping only p dimenion in the tranformed pae. The tranformation play the role of making the lae a eparable a poible through p dimenion only, allowing the afe removal of the remaining n p dimenion; under a Gauian laonditional ditribution aumption, thi amount to giving the ame la-onditional mean and ovariane matrie to thee n p dimenion. Moreover, the projetion i omputed o that the likelihood of the original data i a high a poible. A ummary of the maximum-likelihood approah appear below. Eah la-onditional ditribution in the original pae i aumed to be Gauian. The data are tranformed a y j = Θx j, j = 1,...,, where Θ i an n n invertible matrix. The la-onditional ditribution in the tranformed pae are Gauian with parameter µ = µ, µ n p µ,1,..., µ,p, µ p+1,..., µ n T, Σ 0 Σ =. 0 Σn p ote that only the firt p dimenion are ueful in diriminating between the lae. The objetive in the etimation of Θ i the maximization of the log-likelihood of the original data: L{x 1, 1,..., x, } = log φx j ; µ, Σ. Conditioned on la, the relationhip between the pdf of x and y = Θx an be eaily etablihed φx; µ, Σ = Θ φθx; µ, Σ = Θ φ Θ x; µ, Σ φ n p Θ n p x; µ n p, Σ n p

3 where Θ Θ = Θ n p. Thu, the log-likelihood of the training data i given by log Θ logπn + 1 Θ x j µ n p 1 [ T Σ 1 Θ x j µ ] Θ n p x j µ n p T Σ n p 1 Θ n p x j µ n p whih i maximized when µ = 1 Θ x j = Θ µ Σ = 1 Θ x j µ Θ x j µ T = Θ Σ Θ T 3 µ n p = 1 Θ n p x j = Θ n p µ 4 Σ n p = 1 Θ n p x j µ n p Θ n p x j µ n p T = Θ n p ΣΘ n p T, 5 where µ, Σ are the global mean and ovariane of the data, repetively. Subtituting thee value in the expreion for the log-likelihood of the training data, it beome L {x 1, 1,..., x, } = log Θ logπn log Θ Σ Θ T p log Θn p ΣΘ n p n p = log Θ log Θ Σ Θ T log Θn p ΣΘ n p n logπe 6 Expreion 6 i the objetive funtion of Heteroedati Linear Diriminant Analyi HLDA, introdued by Kumar and Andreou [6]. The maximizing Θ annot be given in loed form, and a teepet-deent algorithm i needed for it omputation. However, a Gale point out in [9], in the peial ae where the projeted per-la Gauian are ontrained to have diagonal ovariane matrie, there i a very effiient algorithm for the maximization of 6. Furthermore, a i hown in the next ubetion, when the per-la Gauian ditribution are ontrained to have equal ovariane matrie, maximization of 6 i equivalent to maximization of 1. C. Interpretation of LDA a Maximum-Likelihood Projetion under a Contraint of Equal Per-la Covariane Under a ontraint of equal per-la ovariane, Σ i ontant equal, ay, to Σ 1. The log-likelihood of the training data then beome log Θ logπn 1 Θ x j µ n p 1 1 T Σ 1 1 Θ x j µ Θ n p x j µ n p T Σ n p 1 Θ n p x j µ n p whih i maximized by the ame expreion, 4 and 5, but with 3 replaed by Σ 1 = 1 Θ x j µ Θ x j µ T = Θ WΘ T 7 intead of 3, where W i the average within-la ovariane, defined earlier. Then, the log-likelihood beome L {x 1, 1,..., x, } = log Θ n logπe log Θ WΘ T log Θn p ΣΘ n p 8 Differentiating 8 with repet to Θ e.g., uing formula found in [10] for omputing derivative with repet to matrie and etting the matrix reult to zero, it turn out that the Θ whih maximize 8 atifie the ondition Θ WΘ n p T = 0 and Θ n p ΣΘ T = 0 9 Auming that W i non-ingular, and etting Θ = ΨW 1/, it an be proved that ondition 9 are equivalent to the following two ondition Ψ and Ψ n p are orthogonal 10 Ψ n p W 1/ ΣW 1/ Ψ T = 0, 11 whih are imultaneouly atified when the row of Ψ onit of the orthogonal eigenvetor of W 1/ ΣW 1/ or, equivalently, the orthogonal eigenvetor of W 1/ BW 1/, by virtue of the fat that Σ = W + B. Subtituting thi value of Ψ into 8, the log-likelihood beome logπen W n log1 + ν i 1 i=p+1 where ν i i the i-th eigenvalue of W 1/ BW 1/. Thu, to maximize the likelihood, it uffie to hooe a the p row of Ψ the eigenvetor orreponding to the maximum eigenvalue of W 1/ BW 1/, and then multiply by W 1/ to obtain Θ. It i now obviou that the maximum likelihood etimate of Θ i equal to the LDA olution Θ LDA given earlier.

4 D. Multiple LDA A natural extenion to the HLDA projetion method i to generate multiple tranform intead of a ingle global tranform. Thi etion deribe one way of doing that, alled Multiple LDA MLDA [7]. To motivate the need for uh an extenion, onider the ae where ome of the la-onditional Gauian have mean whih are arbitrarily far from eah other, while ome onfuable lae are uffiiently loe to eah other. In the impler ae where the ovariane matrie are all the ame, it i eay to ontrut an example uh that the loed-form olution of LDA yield a projetion to a lower-dimenional pae whih doe not offer any diriminability between the onfuable lae; the between matrix i jut dominated by the tatiti of the well-eparated lae. On the other hand, having multiple tranform, eah omputed from a group of lae, an mitigate thi problem. In MLDA, a la grouping ha to be peified firt: C lae are divided into S group S C, with eah la being aigned a group label {1,..., S}. ext, the objetive i to etimate a tranformation Θ for eah group in uh a way that all the dirimination information i kept in the firt p dimenion. Uing the notation introdued earlier, we have Θ = Θ Θ n p, where the lat n p dimenion are tranformed independently of the la grouping. A before, the projetion are omputed o that the likelihood of the original Gauian data i a high a poible. The maximum-likelihood approah an be ummarized a follow: Eah la-onditional ditribution in the original pae i aumed to be Gauian. The data are tranformed through y j = Θ x j, j = 1,...,, where Θ i a n n invertible matrix and = j. A with HLDA, the la-onditional ditribution in the tranformed pae are Gauian with parameter µ and Σ, whih are dependent on the la only through the firt p omponent. The objetive in the etimation of Θ i the maximization of the log-likelihood of the original data: L{x 1, 1,..., x, } = log φx j ; µ, Σ. 13 := Conditioning on la, the relationhip between the pdf of the original data x and the tranformed data y = Θ x, =, an be etablihed a φx; µ, Σ = Θ φθ x; µ, Σ = Θ φ Θ x; µ, Σ φ n p Θ n p x; µ n p, Σ n p. Thu in the ae of multiple tranform, the log-likelihood of the training data i given by logπn + 1 Θ x j µ n p 1 [ log Θ T Σ 1 Θ x j µ ] Θ n p x j µ n p T Σ n p 1 Θ n p x j µ n p ote that Θ n p doe not have a ubript, meaning that it i retrited to be the ame for all. The log-likelihood of the training data i maximized when µ = 1 Σ = 1 Θ x j = Θ µ Θ x j µ = Θ Σ Θ T µ n p = 1 Θ n p x j = Θ n p µ Θ x j µ Σ n p = 1 Θ n p x j µ n p Θ n p x j µ n p T = Θ n p ΣΘ n p T, where µ, Σ are the global mean and ovariane of the data, repetively. Subtituting thee value in the expreion for the log-likelihood of the training data give L {x 1, 1,..., x, } = = log Θ logπn log Θ Σ Θ T p log Θn p ΣΘ n p n p log Θ log Θ Σ Θ T log Θn p ΣΘ n p n logπe 14 Thu expreion 14 i the objetive funtion for Multiple LDA. Comparing with the objetive funtion of HLDA 6, the differene lie in having multiple tranform for the p dimenion of the tranformed data. One again, the maximizing Θ annot be given in loed form. Even in the peial ae where the projeted perla Gauian are ontrained to have diagonal ovariane matrie, the implifiation ued in [9] annot be ued T

5 to etimate Θ n p ; intead a ewton-baed optimization heme an be ued. IV. EXPERIMETAL RESULTS Here we aim to tet the projetion heme deribed above under variou ondition. For eah peified ondition, 100 data et of 15 dimenional full ovariane Gauian data are generated for 5 lae, with eah data et ontaining 1000 training ample and 000 teting ample for eah la. For eah data et a projetion i trained uing LDA, HLDA or MLDA to projet the original 15 dimenional data into 3 dimenional pae, and then the reulting lower dimenional tet data are laified by the tatiti obtained from the orreponding training data. In the ae of MLDA when there i more than one poible grouping of the lae, we hoe the grouping that give the bet performane on the training data. The average error rate of the 100 data et are then reported for eah ondition. The firt et of experiment i deigned to ompare the performane of LDA, HLDA and MLDA under different degree of la overlap in the original 15 dimenional pae, whih i refleted by the Baye error and approximated by the laifiation error rate in the original 15 dimenional pae. Five ondition are deignated with the degree of overlap ranging from almot omplete overlap to well eparated, with ondition 1 orreponding to the mot onfuable dataet. The error rate for eah ondition and projetion heme are preented in Table I. Average Error Rate % Condition Approximate Baye Error LDA projetion HLDA projetion Bet MLDA projetion S = group S = 3 group S = 4 group S = 5 group S {1,..., 5} group TABLE I COMPARISO UDER DIFFERET OVERLAP CODITIOS. A een in the table, MLDA alway outperform HLDA, whih alway outperform LDA. Alo note that MLDA give a more ignifiant relative improvement over HLDA when the original data are well eparated ondition 5. But even at the data-et level, MLDA i uperior: Figure 1 how that MLDA reult in a lower error rate for eah one of the 100 experiment ondition. Another et of experiment i ontruted to invetigate how the ize of training data affet the performane of the projetion heme. The reult are hown in Table II with eah olumn orreponding to the perentage of the 1000 ample per la ued for training: while MLDA till perform bet, it performane i the mot affeted by the lak of training data, while LDA i the leat affeted. Fig. 1. HLDA and MLDA Error Rate for all data et of Condition. Average Error Rate % Condition Training Data Size 0% 50% 100% Approximate Baye Error LDA projetion HLDA projetion Bet MLDA projetion S = group S = 3 group S = 4 group S = 5 group S {1,..., 5} group TABLE II COMPARISO WITH DIFFERET AMOUTS OF TRAIIG DATA. V. COCLUDIG REMARKS From our experiment we oberved that under the variou ondition we ontruted, MLDA alway give the bet performane. Higher dimenional projetion were alo examined, and the ame trend hold. At the ame time, how muh MLDA an improve over HLDA and LDA i determined by the harateriti of the original data. REFERECES [1] F. Jelinek, Statitial Method for Speeh Reognition, MIT Pre, [] M. Gale and S. Young, The Appliation of Hidden Markov Model in Speeh Reognition, OW Publiher, 008. [3] A. P. Dempter,. M. Laird, and D. B. Rubin, Maximum likelihood from inomplete data via the EM algorithm, Journal of the Royal Statitial Soiety, Serie B Methodologial, vol. 39, pp. 1 38, [4] R. A. Fiher, The ue of multiple meaurement in taxonomi problem, Annal of Eugeni, vol. 7, pp , [5] C. R. Rao, Linear tatitial inferene and it appliation, Wiley, ew York, [6]. Kumar and A. G. Andreou, Heteroedati diriminant analyi and redued rank HMM for improved peeh reognition, Speeh Communiation, vol. 6, pp , [7] M. Gale, Maximum likelihood multiple ubpae projetion for hidden markov model, IEEE Tranation on Speeh and Audio Proeing, vol. 10, no., pp , February 00. [8]. Campbell, Canonial variate analyi a general formulation, Autralian Journal of Statiti, vol. 6, pp , [9] M. Gale, Semi-tied ovariane matrie for hidden Markov model, IEEE Tranation on Speeh and Audio Proeing, vol. 7, no. 3, pp. 7 81, May [10] S. R. Searle, Matrix Algebra ueful for Statiti, Wiley, ew York, 198.