On the Convergence of Bound Optimization Algorithms

Transcription

1 On the Convergence of ound Optmzaton lgorthms Ruslan Salakhutdnov Sam Rowes Unversty of Toronto 6 Kng s College Rd, MS 3G4, Canada rsalakhu,rowes@cs.toronto.edu Zoubn Ghahraman Gatsby Computatonal Neuroscence Unt Unversty College London 17 Queen Square, London WC1N 3R, UK zoubn@gatsby.ucl.ac.uk bstract Many practtoners who use EM and related algorthms complan that they are sometmes slow. When does ths happen, and what can be done about t? In ths paper, we study the general class of bound optmzaton algorthms ncludng EM, Iteratve Scalng, Non-negatve Matrx Factorzaton, CCCP and ther relatonshp to drect optmzaton algorthms such as gradentbased methods for parameter learnng. We derve a general relatonshp between the updates performed by bound optmzaton methods and those of gradent and second-order methods and dentfy analytc condtons under whch bound optmzaton algorthms exhbt quas-newton behavor, and under whch they possess poor, frst-order convergence. ased on ths analyss, we consder several specfc algorthms, nterpret and analyze ther convergence propertes and provde some recpes for preprocessng nput to these algorthms to yeld faster convergence behavor. We report emprcal results supportng our analyss and showng that smple data preprocessng can result n dramatcally mproved performance of bound optmzers n practce. 1 ound Optmzaton lgorthms Many problems n machne learnng and pattern recognton ultmately reduce to the optmzaton of a scalar valued functon LΘ) of a free parameter vector Θ. For example, n supervsed and unsupervsed probablstc modelng the objectve functon may be the condtonal) data lkelhood or the posteror over parameters. In dscrmnatve learnng we may use a classfcaton or regresson score; n renforcement learnng an average dscounted reward. Optmzaton may also arse durng nference; for example we may want to reduce the cross entropy between two dstrbutons or mnmze a functon such as the ethe free energy. ound optmzaton O) algorthms take advantage of the fact that many objectve functons arsng n practce have a specal structure. We can often explot ths structure to obtan a bound on the objectve functon and proceed by optmzng ths bound. Ideally, we seek a bound that s vald everywhere n parameter space, easly optmzed, and equal to the true objectve functon at one or more) ponts). general form of a bound maxmzer whch teratvely lower bounds an objectve functon LΘ) s gven below: General ound Optmzer for maxmzng LΘ): ssume: GΘ, Ψ) such that for any Θ and Ψ : 1. GΘ, Θ ) = LΘ ) & LΘ) GΘ, Ψ ) Ψ Θ 2. arg max Θ GΘ, Ψ ) can be found easly for any Ψ. Iterate: Θ t+1 = arg max Θ GΘ, Θ t ) Guarantee: LΘ t+1 ) = GΘ t+1, Θ t+1 ) GΘ t+1, Θ t ) GΘ t, Θ t ) = LΘ t ) ound optmzers do nothng more than coordnate ascent n the functonal GΘ, Ψ), alternatng between maxmzng G wth respect to Ψ for fxed Θ and wth respect to Θ for fxed Ψ. These algorthms enjoy a strong guarantee; they never worsen the objectve functon. Many popular teratve algorthms are bound optmzers, ncludng the EM algorthm for maxmum lkelhood learnng n latent varable models2, teratve scalng IS) algorthms for parameter estmaton n maxmum entropy models1, non-negatve matrx factorzaton NMF)3 and the recent CCCP algorthm for mnmzng the ethe free energy n approxmate nference problems12. In ths paper we explore two questons of theoretcal and practcal nterest: when wll bound optmzaton be fast or slow relatve to other standard approaches, and what can be done to mprove convergence rates of these algorthms when they are slow? 2 Convergence ehavor and nalyss How large are the steps that bound optmzaton methods take? ny bound optmzer mplctly defnes a mappng: M : Θ Θ from parameter space to tself, so that Θ t+1 = MΘ t ). If terates Θ t converge to a fxed pont Θ, then Θ = MΘ ). If MΘ) s contnuous and dfferentable, we can Taylor expand t n the neghborhood of

2 the fxed pont Θ : Θ t+1 Θ M Θ )Θ t Θ ) 1) where M Θ ) = M Θ=Θ. Snce M Θ ) s typcally nonzero, a bound optmzer can essentally be seen as a lnear teraton algorthm wth a convergence rate matrx M Θ ). Intutvely, M Θ ) can be vewed as an operator that forms a contracton mappng around Θ. In general, we would expect the Hessan 2 LΘ) 2 Θ=Θ to be negatve semdefnte, or negatve defnte, and thus the egenvalues of M Θ ) to all le n, 1 or, 1) respectvely 4. Exceptons to the convergence of the bound optmzer to a local optmum of LΘ) occur f M Θ ) has egenvalues whose magntudes exceed unty. Near a local optmum, ths matrx s related to the curvature of the functonal GΘ, Ψ): lm M Θ t ) = 2 Θ t Θ G Θ, Ψ ) 2 G Θ ) 1 2) where we defne the mxed partals and Hessan as: 2 G Θ, Ψ ) 2 GΘ,Ψ) Ψ Θ = Θ T Ψ = Θ 2 GΘ ) Θ = Θ Ψ = Θ 2 GΘ,Ψ) T We assume we can easly fnd arg max Θ GΘ, Ψ), and thus 2 G Θ ) s negatve defnte nvertble). Proof sketch of eq 2): y performng Taylor seres expanson of GΘ 2, Θ 1 )= GΘ,Θ1 ) Θ=Θ 2 around Θ, Θ ), we have: GΘ 2, Θ 1 ) = GΘ, Θ ) + Θ 2 Θ ) T 2 GΘ ) + Θ 1 Θ ) T 2 GΘ, Ψ ) Substtutng Θ t for Θ 1, and MΘ t ) for Θ 2 gves = MΘ t ) Θ ) T 2 GΘ ) + Θ t Θ ) T 2 GΘ, Ψ ) ssumng that hgher order terms are neglgble, n the lmt, Θ = MΘ ) and = lm Θ t Θ M Θ t )) 2 GΘ ) + 2 GΘ, Ψ ). ) What drectons do bound optmzers move n parameter space? For most objectve functons, the O step Θ t+1) Θ t) n parameter space and true gradent vector L Θ t ) = LΘ) Θ=Θt can be trvally related by a transformaton matrx P Θ t ), that changes at each teraton: 3) 4) Θ t+1) Θ t) = P Θ t ) L Θ t ) ) Under certan condtons, ths transformaton matrx P Θ t ) s guaranteed to be postve defnte wth respect to any gradent. In partcular, f C1: GΘ, Θ t ) s well-defned, and dfferentable everywhere n Θ; and C2: for any fxed Θ t Θ t+1), along any drecton that passes through Θ t+1, GΘ, Θ t ) has only a sngle crtcal pont n ts frst argument, located at the maxmum Θ t+1 ; then LΘ t )P Θ t ) L Θ t ) > Θ t 6) The second condton may seem very strong, however, t s satsfed n many practcal cases. For example, for the EM algorthm, t s satsfed whenever the M-step has a sngle unque soluton n partcular, t holds for exponental famly models due to concavty of GΘ, Θ t )); for GIS, NMF, CCCP, and many others, t s satsfed due to concavty of GΘ, Θ t ) although C2 does not mply concavty). Proof sketch of eq 6): For GΘ t )Θ t+1) Θ t ), we note that GΘ t ) = GΘ,Θt ) Θ=Θ t s the drectonal dervatve of functon GΘ, Θ t ) n the drecton of Θ t+1) Θ t. C1 and C2 together mply that ths quantty s postve, otherwse by the Mean Value Theorem C1) GΘ, Θ t ) would have a crtcal pont along some drecton, located at a pont other than Θ t+1 C2). y usng the dentty LΘ t ) = GΘ,Θt ) Θ=Θ t, we have LΘ t )P Θ t ) LΘ t ) = GΘ t )Θ t+1) Θ t ) >. ) The mportant consequence of the above analyss s that when the bound functon has a unque optmum wrt ts frst argument, O has the appealng qualty of always takng a step Θ t+1) Θ t havng postve projecton onto the true gradent of the objectve functon LΘ t ). Ths makes O smlar to a frst order method operatng on the gradent of a locally reshaped lkelhood functon. For maxmum lkelhood learnng of a mxture of Gaussans model usng the EM-algorthm, ths postve defnte transformaton matrx P Θ t ) was frst descrbed by Xu and Jordan11. We have extended ther results by dervng the explct form of the transformaton matrx for several other latent varables models such as Factor nalyss F), Probablstc Prncpal Component nalyss PPC), mxture of PPCs, mxture of Fs, and Hdden Markov Models 8; we have also derved the general form of P Θ t ) matrx for exponental famly models n terms of natural parameters. Here we further study the structure of the transformaton matrx P Θ t ) and relate t to the convergence rate matrx M. Our man result s that when the dervatve s small M has small egenvalues), the transformaton matrx approaches the negatve nverse Hessan and bound optmzaton behaves lke a second-order Newton method. In partcular, n the neghborhood of a local optmum Θ : lm Θ t Θ P Θt ) = I M Θ ) 1 SΘ ) 7) where SΘ ) = 2 LΘ) 2 Θ=Θ s the Hessan of the objectve functon. We assume that P Θ) and MΘ) are dfferentable and that SΘ ) 1 exsts. Proof sketch of eq 7): Takng negatve dervatves of ) wrt Θ t yelds I M Θ t ) = P Θ t ) LΘ t ) P Θ t )SΘ t ) where M Θ t ) = t+1 / t j s the nput-output dervatve matrx for the O mappng and P Θ t ) = P Θt ) Θ=Θt s the tensor dervatve of P Θ t ) wth respect to Θ t. In the lmt, near a fxed pont, the frst term wll vansh snce the gradent s gong to zero assumng P Θ t ) does not become nfnte); the equalty 7) readly follows. ) Ths shows that the nature of the quas-newton behavor s controlled by the convergence matrx M Θ ). When the matrx M has small egenvalues, then near a local optmum bound optmzaton may exhbt quas-newton con-

3 1 8.4 GRDIENT EM NEWTON 6 4 µ 1 µ µ 1 µ GRDIENT 9. NEWTON EM Fgure 1: Contour plots of the lkelhood functon LΘ) for MoG examples usng well-separated upper panels) and not-well-separated lower panels) one-dmensonal data sets. xes correspond to the two means. The dashdot lne shows the drecton of the true gradent LΘ), the sold lne shows the drecton of P Θ) LΘ) and the dashed lne shows the drecton of S) 1 LΘ). Rght panels are blowups of dashed regons on the left. The numbers ndcate the log of the l 2 norm of LΘ). For the well-separated case, n the vcnty of Θ, vectors P Θ) LΘ) and S) 1 LΘ) become dentcal. vergence behavor. Ths s also true n plateau regons where the gradent s very small even f they are not near a local optmum. We can examne the structure of ths matrx and ts egenvalues, or the rato of ts two top egenvalues. In partcular, f the top egenvalue of M Θ ) tends to zero, then O becomes a true Newton method, rescalng the gradent by exactly the negatve nverse Hessan. Θ t+1 = Θ t SΘ t ) 1 L Θ t ) 8) s the egenvalues tend to unty, O takes smaller and smaller stepszes, gvng poor, frst-order convergence. 3 Common ound Optmzers 3.1 Expectaton-Maxmzaton EM) We now consder a partcular bound optmzer, the popular Expectaton-Maxmzaton EM) algorthm, and derve specfc cases of the results above for models whch use EM to adjust ther parameters. To begn, consder a probablstc model of observed data x whch uses latent varables y. For any value of Ψ, t can be easly verfed that the followng dfference of two terms s a lower bound on the lkelhood: GΘ, Ψ) = QΘ, Ψ) HΨ, Ψ) = py x, Ψ) ln px, y Θ)dy py x, Ψ) ln py x, Ψ)dy The log lkelhood functon can be wrtten as: LΘ) = ln px Θ) = py x, Θ) ln px Θ)dy = GΘ, Θ) GΘ, Ψ) Ψ y 2), we can establsh: 1 2 G Θ ) = 2 QΘ, Θ ) 2 Θ=Θ 2 GΘ, Ψ ) = 2 HΘ, Θ ) 2 Θ=Θ and therefore we have an expresson for M Θ ): MΘ) Θ=Θ = 2 HΘ,Θ ) 2 Θ=Θ 2 QΘ,Θ ) 2 Θ=Θ 1 Ths can be nterpreted as the rato of mssng nformaton to complete nformaton near the local optmum 2,. Notce that the curvature of the orgnal bound functon appears as one of the terms n the rato. ccordng to 7), n the neghborhood of a soluton for suffcently large t): P Θ t ) I 2 H 2 ) 2 Q 2 ) 1 Θ=Θ t SΘ t ) 1 The nterpretaton of ths result s ntutve and well known: When the mssng nformaton s small compared to the complete nformaton, EM exhbts quas-newton behavor and enjoys fast, typcally superlnear, convergence n the neghborhood of Θ. If the fracton of mssng nformaton approaches unty, the egenvalues of the frst term above approach zero and EM wll exhbt extremely slow convergence. The above analyss gves a formal explanaton applcable to any latent varable model) of ths behavour. Fgure 1 llustrates these results n the case of fttng a mxture of Gaussans model to well-clustered and not-wellclustered data. Many other models also show ths same 1 For further detals refer to 4

4 effect; for example, when Hdden Markov Models or ggregate Markov Models 9 are traned on very structured sequences, EM exhbts quas-newton behavor, n partcular when the state transton matrx s sparse and the output dstrbutons are almost determnstc at each state. 3.2 Generalzed Iteratve Scalng GIS) In ths secton we consder the Generalzed Iteratve Scalng algorthm 1, wdely used for parameter estmaton n maxmum entropy models. Its goal s to determne the parameters Θ of an exponental famly 1 dstrbuton px Θ) = ZΘ) exp ΘT F x)) such that certan generalzed margnal constrants are preserved: x px Θ )F x) = x px)f x), where ZΘ) s the normalzng factor, px) s a gven emprcal dstrbuton and F x) = f 1 x),..., f d x), 1 T s a gven feature vector on the nputs. We nclude the constant, or bas feature.) The GIS algorthm requres that f x) > but we wll not requre f x) = 1)6. The log-lkelhood s: LΘ) = x px) ln px Θ) = x px)θt F x) ln ZΘ) We note that ln ZΘ) ZΘ)/ZΨ) + ln ZΨ) 1 for any Ψ, and exp Θ f x) f x) exp Θ + 1 f x), wth f x) 1. Defnng s = max x f x), we construct a lower bound: f x) s LΘ) x px) Θ f x) ln ZΨ) + px Ψ) f x) exp sθ Ψ ) = GΘ, Ψ) s x Ths lower bound has the useful property that ts maxmzaton s decoupled across the parameters Θ. The GIS algorthm s then gven by: Θ t+1 = Θ t + 1 s ln x px)f x) x px Θt )f x) Defne F Θ ) x px Θ )F x) to be the mean of the feature vectors, DΘ ) dag F Θ ) to be the correspondng dagonal matrx, and CovΘ ) to be covarance of the feature vectors under model dstrbuton px Θ ). We can compute second order statstcs usng 2): 2 G Θ ) = s dag F Θ ) = sdθ ) 2 G Θ, Ψ ) = s dag F Θ ) x px Θ )F x)f x) T F Θ ) F Θ ) T = sdθ ) CovΘ ) ccordng to 7), n the neghborhood of a soluton for suffcently large t), the step GIS takes n parameter space and true gradent are related by the matrx: 1 1 P Θ t ) s CovΘt )DΘ t ) 1 SΘ t ) Due to the concavty of GΘ, Ψ ) for any fxed Ψ, the step a GIS algorthm takes n parameter space always has postve projecton onto the true gradent of the objectve functon. The convergence rate matrx M Θ ) s of the form: MΘ) Θ=Θ = I 1 s CovΘ )DΘ ) 1 9) and depends on the covarance and the mean of the feature vectors. We can nterpret ths result as follows: when feature vectors become less correlated and closer to the orgn, GIS exhbts faster convergence n the neghborhood of Θ. If features are hghly dependent, then GIS wll exhbt extremely slow convergence. 3.3 Non-Negatve Matrx Factorzaton NMF) Gven a non-negatve matrx V, the NMF algorthm3 tres to fnd matrces W and H, such that V W H. Posed as an optmzaton problem, we are nterested n mnmzng a dvergence LW, H) = DV W H), subject to W, H) elementwse: V ln V W H) V + W H) LW, H) = We use ln c W ch cj c α c, c) ln WcHcj α c,c) where α a, b) = Wa t Ht bj / r W r t Ht rj, so that α c, c) sum to one. Defnng Θ = W, H) and Ψ = W t, H t ), we can construct the upper bound on the cost functon: LΘ) V ln V V + W c H cj 1) c V α c, c) ln W ch cj = GΘ, Ψ) α c, c) c One can now compute second order statstcs usng 2). In the appendx we derve the explct form of the convergence rate matrx M. We also note that the convergence matrx of NMF much resembles the convergence matrx of GIS, snce both algorthms make use of the bound that comes from Jensen s nequalty. 3.4 Concave-Convex Procedure CCCP) CCCP 12 optmzer seeks to mnmze an energy functon EΘ), whch can be decomposed nto a convex E vex Θ) and a concave E cave Θ) functon: EΘ) = E vex Θ) + E cave Θ) 11) CCCP algorthm s gven by: E vex Θ t+1 ) = E cave Θ t ) It s easy to see that CCCP belongs to the class of bound optmzaton algorthms, and therefore can be analyzed as a frst order teratve algorthm. Its bound functon s: EΘ) E vex Θ) + E cave Ψ) + Employng 2), we have: ) Θ Ψ) T E cave Ψ) = GΘ, Ψ) 2 G Θ ) = 2 E vex Θ) T Θ=Θ 2 G Θ, Ψ ) = 2 E cave Ψ) Ψ Ψ T The convergence rate matrx s gven by: M Θ ) = 2 E caveψ) Ψ Ψ T Ψ=Θ Ψ=Θ 2 E vexθ) T Θ=Θ 1 whch can be nterpreted as a rato of concave curvature to convex curvature. ccordng to 7) n the neghborhood of

5 a soluton for suffcently large t) the gradent and step are related by: P Θ t ) ) ) 1 1 I 2 E cave 2 E vex 2 2 Θ=Θ t SΘ t ) Of course, the step CCCP takes n parameter space has postve projecton onto the true gradent of the orgnal energy functon EΘ). The above vew of CCCP has an nterestng nterpretaton: If the concave energy functon has small curvature compared to the convex energy term n the neghborhood of Θ, CCCP wll exhbt a quas-newton behavor and wll possess fast, typcally superlnear convergence. s the fracton of concave-convex curvature approaches one, CCCP wll exhbt extremely slow, frst order convergence behavor. Fgure 4 llustrates exactly such an example. 4 Improvng Convergence Rates The above analyss helped to answer the queston: when and why wll bound optmzers converge slowly? They can also help to answer the more practcal queston: what can we do to speed up convergence? In the case of EM, t s possble to estmate the key quantty controllng convergence fracton of mssng nformaton) and swtch to drect gradent-based) optmzaton when we predct slow behavor of EM. We have expermented wth such a hybrd approach wth some success7. For other bound optmzers, smlar hybrd algorthms are possble. ut there s another, ntrgung approach to mprovng convergence speed: modfy the orgnal nput to the algorthms based on our analyss of convergence rates. In the case of GIS ths nvolves transformng features, n the case of NMF, ths requres scalng and translatng data vectors, and for CCCP ths comes down to desgnng dfferent convexconcave decompostons of the objectve. These nput modfcatons do not change the fnal results of the algorthms; they only change the convergence propertes. egnnng wth GIS, we can show that translatng feature vectors to brng them closer to the orgn and decorrelatng whtenng) them both speed up convergence. Homogeneously rescalng all features by a sngle constant does not affect convergence.) In partcular, the optmal translaton of features s gven by F new x) = F x) V wth V = mn x f x), and the optmal lnear transformaton F new s that whch makes CovΘ ) T equal to dentty matrx, takng nto account the bas term, or a feature that s a constant. We provde sketch proofs of both results n the appendx.) Of course, the covarance n the second condton cannot be evaluated untl the optmal parameters are known, but t can be approxmated by usng the sample covarance of features on the tranng set. For NMF, smlar to GIS, we can show that translatng data vectors to brng them closer to the orgn speeds up convergence, whereas homogeneously rescalng all data by a sngle constant does not affect convergence. For CCCP, t s well-known that any energy functon wth bounded curvature has many convex-concave decompostons but no clear prncple for fndng a good one has been known. Our analyss provdes gudance n ths regard: we should mnmze the rato of curvatures between the convex and concave parts of the energy. In the next secton we llustrate that approprate preprocessng of the nput to these varous bound optmzaton algorthms does result n a much faster rate of convergence. Expermental Results We now present emprcal results to support the valdty of our analyss for several bound optmzaton algorthms. We frst apply EM to learnng the parameters of two latent varable models: Mxtures of Gaussans MoG) and Hdden Markov Models HMM). We then analyze and apply Iteratve Scalng IS) to a logstc regresson model. Next, we show the effect of data translaton on the convergence propertes of NMF. Fnally, we fnsh by descrbng and analyzng the effect of varous energy functon decompostons on the convergence behavor of the CCCP algorthm. Though not shown, we confrmed that the convergence results presented below do not vary sgnfcantly for dfferent random ntal startng ponts n the parameter space. Frst, consder a mxture of Gaussans MoG) model. In ths model the proporton of mssng nformaton corresponds to how well or not-well the data s separated nto dstnct clusters. We therefore consdered two types of data sets, a well-separated case and a not-wellseparated case n whch the data overlaps n one contguous regon. s predcted by our analyss, n the wellseparated case, n the vcnty of the local optmum Θ the drectons of the vectors P Θ) L Θ) and S) 1 L Θ) become dentcal fg. 1), showng that EM wll have quas- Newton convergence behavor. In not-well-separated case, due to the large proporton of mssng nformaton, these drectons are sgnfcantly dfferent and EM possesses poor, frst-order convergence behavor. We also appled the MoG model to cluster a set of, 8 8 greyscale pxel mage patches. 2 Fgure 2 dsplays the convergence behavor of EM for M= and M= mxture components. The expermental results reveal, that wth fewer mxture components, EM converges quckly to a local optmum, snce the components generally model the data wth farly dstnct, non-contguous clusters. s the number of mxtures components ncreases, clusters overlap n contguous regons, creatng a relatvely hgh proporton of mssng nformaton. In ths case the convergence of EM slows by several orders of magntude. We then appled EM to tranng Hdden Markov Models HMMs). Mssng nformaton n ths model s hgh when the observed data do not well determne the underlyng 2 The data set used was the mlog data set publcly avalable at ftp://hlab.phys.rug.nl/pub/samples/mlog

6 Log Lkelhood + Const EM: Mxture of Gaussans Log Lkelhood + Const EM: Hdden Markov Models Structured Sequence : ECDEEC... Unstructured Sequence : CEDCEDDC Log Lkelhood + Const x 1 3 EM: Mxture of Gaussans Component Component Fgure 2: Learnng curves of EM algorthm for two models: MoG and HMM. Dfferent data sets are shown on the same plots for convenence. The teraton number s shown on the horzontal axs, and log-lkelhood s shown on the vertcal axs wth the zerolevel lkelhood correspondng to the convergng pont of the EM algorthm. For well-separated and structured data ), EM possesses quas-newton convergence behavor. EM n ths case converges n 1-1 teratons wth stoppng crteron: LΘ t+1 ) LΘ t )/abslθ t+1 )) < 1 1. For overlappng, alased data ), EM posses poor, frst-order convergence. Rght panel dsplays convergence behavor of EM by fttng component as opposed to component MoG model on the same data set of gray mage patches. Condtonal Log Lkelhood + Const , C Logstc Regresson Feature Feature C Iteraton Number Feature Feature Condtonal Log Lkelhood + Const Z Y Logstc Regresson X Feature Y X Feature Z X Interaton Number Feature Fgure 3: Learnng curves left panels) of Iteratve Scalng algorthm for logstc regresson model, showng the effect that translaton and whtenng of the feature vectors have on the IS convergence behavor, wth letters correspondng to the respectve data sets. Top panels show an experment wth 2, 2-dmensonal feature vectors drawn from standard normal, bottom panels dsplay an dentcal experment wth 2, feature vectors drawn from normal wth orented covarance. Top, rght panel shows that scalng feature vectors by constant does not affect the convergence of IS. state sequence gven the parameters). We therefore generated two synthetc data sets from a -state HMM, wth an alphabet sze of characters. The frst data set alased sequences) was generated from a HMM where output parameters were set to unform values plus some small nose ɛ N, 1I). The second data set structured sequences ) was generated from a HMM wth sparse transton and output matrces. Fgure 2 shows that for the very structured data, EM performs well and exhbts second order convergence n the vcnty of the local optmum. For the ambguous or alased data, EM posses extremely slow, frst-order convergence behavor. Ths analyss may also shed lght on why hard-clusterng algorthms such as k-means and Vterb style E-steps for HMMs appear to have faster convergence than ther softer cousns: they suppress the mssng nformaton. Feature 1 To confrm our analyss of GIS, we appled teratve scalng algorthm to a smple 2-class logstc regresson model: py = ±1 x, w) = 1/1 + exp yw T x)) followng 6. In our frst experment, N feature vectors of dmensonalty d were drawn from normal: x N, 2I d ), wth the true parameter vector w beng randomly chosen on the surface of the d-dmensonal sphere wth radus 2. To make features postve, the data set was modfed by addng 2 to all feature values. Fgure 3 shows that for N = 2 and d = 2, nave IS, that runs on the orgnal unpreprocessed features, takes over 2 teratons to converge. When feature vectors are translated closer to the orgn, IS converges to exactly the same maxmum lkelhood soluton, but beats nave IS by a factor of almost twelve. Our second experment was smlar, but feature vectors of dmensonalty d were drawn from a Gaussan wth or-

7 Dvergence NMF Iteraton Number Energy Functon Ex) Vex1 + Cave1 CCCP Vex2 + Cave Vex3 + Cave3 Iteraton Number Ex) E vex1 x) E vex3 x) E vex2 x) E cave3 x) E cave1 x) E cave2 x) Fgure 4: Learnng curves of NMF and CCCP algorthms. For NMF, we show the effect that data translaton has on the convergence behavor of NMF n our case black pxels correspond to, whte to 3). pplyng CCCP to mnmze a smple energy functon Ex) = x 4 3x 2 + 2x 2, we dsplay the effect that dfferent energy decompostons left panel) have on CCCP convergence. ented covarance. Fgure 3 shows that for N=2, and d=2, translatng features mproves the convergence of IS by a factor of over 4, whereas translatng and whtenng feature vectors results n speedup by factor of twenty. Smlar results are obtaned f dmensonalty of data s ncreased. Next, we expermented wth the NMF algorthm. Data vectors were drawn from standard normal: x N, I 16 ). To make features postve, the data set was modfed by addng 2 to all data values, formng non-negatve matrx V. We then appled NMF to perform non-negatve factorzaton: V W H. Fgure 4 reveals that nave NMF, that runs on the orgnal unpreprocessed data data set ), takes over 1,3 teratons to converge. Once data vectors are translated closer to the orgn data set ), NMF converges to exactly the same value of the cost functon n about 23 teratons, outperformng nave NMF by a factor of over fve. Fnally, we expermented wth the CCCP algorthm. We consdered a smple energy functon Ex)=x 4-3x 2 +2x-2, whch has many decompostons fg.4). decomposton whch mnmzes the rato of concave-convex curvature s: E cave1 x)=-3x 2-2 and E vex1 x)=x 4 +2x. Other decompostons: E cave2 x)=-13x 2-2 and E vex2 x)=x 4 +1x 2 +2x; E cave3 x)=-9x 4-3x 2-2 and E vex3 x)=1x 4 +2x; clearly ncrease the proporton of concave-convex curvature. In our experment, all runs of CCCP were started from the same ntal pont n the parameter space. Fgure 4 reveals that as the proporton of the local concave-convex curvature ncreases, the convergence rate of CCCP sgnfcantly slows down, by several orders of magntude. 6 Dscusson In ths paper we have analyzed a large class of bound optmzaton algorthms and ther relatonshp to drect optmzaton algorthms such as gradent-based methods. We determned condtons under whch O algorthms exhbt local-gradent and fast quas-newton convergence behavors. ased on ths analyss and nterpretaton, we have also provded some recommendatons for how the nput to these algorthms can be preprocessed to yeld faster convergence. Currently, usng dervaton of an explct form of the convergence rate matrx, we are also workng on dentfyng analytc condtons under whch CCCP possesses fast or extremely slow convergence n mnmzng ethe and Kkuch free energes n approxmate nference problems. Smlar analyss can be appled to other bound optmzaton algorthms; for example Sha et. al. 1 recently ntroduced a multplcatve algorthm for tranng SVMs and provded a convergence analyss based on margns. Our analyss and experments show that n the regme where the convergence rate matrx has large egenvalues, a bound optmzer s lkely to perform poorly. Slow convergence s expected when mssng nformaton s hgh whle learnng wth EM; when feature vectors are hghly dependent whle estmatng parameters wth GIS or NMF; or when the rato of concave-convex curvature s large when mnmzng energy functon wth CCCP. In these cases, one can ether attempt to modfy the basc O algorthms to accelerate them, or nstead employ drect optmzaton algorthms such as conjugate-gradent whch are lkely to have far superor performance. longsde our analyss we have also presented a thrd alternatve: nputs to standard O algorthms can sometmes be preprocessed to speed convergence. cknowledgments Funded n part by the IRIS project, Precarn Canada. References 1 Stephen Della Petra, Vncent J. Della Petra, and John D. Lafferty. Inducng features of random felds. IEEE Transactons on Pattern nalyss and Machne Intellgence, 194):38 393, P. Dempster, N. M. Lard, and D.. Rubn. Maxmum lkelhood from ncomplete data va the EM algorthm wth dscusson). J. of the RS Socety seres, 39:1 38, Danel D. Lee and H. Sebastan Seung. Learnng the parts of objects by non-negatve matrx factorzaton. Nature, 41: , G. J. McLachlan and T. Krshnan. The EM lgorthm and Extensons. Wley, Xao-L Meng and Donald. Rubn. Maxmum lkelhood estmaton va the ECM algorthm: general framework. ometrka, 8: , Tom Mnka. lgorthms for maxmum-lkelhood logstc regresson. Techncal Report 78, Dept. of Statstcs, Carnege Mellon Unversty, 21.

8 7 Ruslan Salakhutdnov, Sam Rowes, and Zoubn Ghahraman. Optmzaton wth EM and Expectaton-Conjugate- Gradent. In To appear n Proc. 2th Internatonal Conf. on Machne Learnng, Ruslan Salakhutdnov, Sam Rowes, and Zoubn Ghahraman. Relatonshp between gradent and EM steps for several latent varable models. rsalakhu/ecg. 9 Lawrence Saul and Fernando Perera. ggregate and mxedorder Markov models for statstcal language processng. In Proceedngs of the Second Conference on Emprcal Methods n Natural Language Processng, pages Fe Sha, Lawrence Saul, and Danel Lee. Multplcatve updates for nonnegatve quadratc programmng n support vector machnes. In dvances n NIPS, volume 1, L. Xu and M. I. Jordan. On convergence propertes of the EM algorthm for Gaussan mxtures. Neural Computaton, 81):129 11, lan Yulle and nand Rangarajan. The convex-concave computatonal procedure CCCP). In dvances n NIPS, volume 13, 21. ppendx Clam 1: Translatng feature vectors closer to the orgn speeds up convergence of GIS. The optmal translaton of features s gven by F newx) = F x) V x wth a vector V contanng elements V = mn x f x). Proof sketch: Consder settng F newx) = F x) V x as above. We have M newθ ) = I 1 s new CovΘ )D newθ ) 1 12) wth D newθ ) = DΘ ) dagv ) see eq. 9), and s new = s V. Let us denote QΘ ) CovΘ )DΘ ) 1, Q newθ ) CovΘ )D newθ ) 1, and λ max) the largest egenvalue of. We can now show that ths translaton forces the top egenvalue of M Θ ) to decrease: λ maxm newθ )) λ maxm Θ )), where we derved 9): M Θ ) = I 1 s QΘ ). Note that: λ maxm newθ )) = 1 λ 1 mn s new Q newθ ) ). Hence, our task reduces to showng: 1 λ mn Q newθ ) ) 1 λ mn s new s QΘ ) ) λ max snewq 1 newθ ) ) λ max sq 1 Θ ) ) 13) Takng nto account that s new obvous by examnng: λ max snewq 1 newθ ) ) = s, the above nequalty s s newλ max DΘ ) dagv ) Cov 1 Θ ) ) sλ max DΘ )Cov 1 Θ ) ) = sλ max Q 1 Θ ) ) It s now clear that the optmal translaton of features s gven by F newx) = F x) V x wth V = mn x f x). Clam 2: Decorrelatng whtenng) feature vectors speeds up convergence of GIS In partcular, the optmal lnear transformaton F newx) = F x) s that whch makes CovΘ ) T equal to dentty matrx. Proof sketch: Consder spectral decomposton: CovΘ ) = W HW T, wth H beng the dagonal matrx of the egenvalues, and W beng the orthogonal matrx of the correspondng egenvectors. Let = W H 1/2 W T. The lnear transformaton becomes F newx) = F x) 3, n whch case CovΘ ) T = I. M newθ ) = I 1 s new CovΘ ) T D newθ ) 1 = I 1 D newθ ) 1 14) s new wth s new = max x F x), and DnewΘ ) = dag x px Θ )F x) = dag F Θ ). We now show that, n general, λ maxm newθ )) λ maxm Θ )). Ths task reduces to showng see eq 13)): λ max snewd newθ ) ) λ max sq 1 Θ ) ). Frst note that: λ max sq 1 Θ ) ) = sλ max DΘ )Cov 1 Θ ) ) On the other sde: = sλ max DΘ ) T ) 1) s newλ max DnewΘ ) ) = s new F Θ ) s new DΘ ) 16) It can also be shown that s new sλ max) = s 2. y usng above facts, slghtly more relaxed bound holds: DΘ ) s new 2 DΘ ) s 2 17) Therefore n general, whtenng feature vectors, pushes down the top egenvalue of the convergence rate matrx, whch accordng to our analyss, results n ts faster rate of convergence. Non-Negatve Matrx Factorzaton: We use a bound on the objectve functon 1) to derve the explct form of the convergence rate matrx M. Defnng Θ = W, H) and Ψ = W t, H t ), we employ 2): 2 GΘ ) Wc W = δ kp k δ cp j 2 GΘ ) = δ cpδ Hcj jl H pl 2 GΘ, Ψ ) W c W kp 2 GΘ, Ψ ) H cj H pl 2 GΘ, Ψ ) W c H pl 2 GΘ, Ψ ) H cj W kp V V = δ k δ cp = δ jl δ cp j H cj W c W c H cj V V = V δ cp α l c, p)) = V kj δ V kj cp α kj c, p)) 2 GΘ ) = δ Wc cp H pl 2 GΘ ) Hcj W = δ kp cp Hcj δ Wc k j Wc δ Hcj jl V V Hcj α c, p) Wc Wc Hcj where we defne = c W ch cj, and δ = 1 f = j; otherwse. The convergence rate matrx M wll be of the form: MΘ) Θ=Θ = 2 GΘ, Ψ ) 2 GΘ ) 1 3 Here we are assumng that the new feature vector Fx) has only postve entres. If Fx) has negatve entres t mght be necessary to decorrelate and add a translaton, whch trades off the advantage of Clam 1 and Clam 2. α c, p))