DIFFUSION ESTIMATION OF MIXTURE MODELS WITH LOCAL AND GLOBAL PARAMETERS

Transcription

1 DIFFUSION ESTIMATION OF MIXTURE MODES WITH OCA AND GOBA PARAMETERS Kaml Dedecus and Vladmíra Sečkárová Insue of Informaon Theory and Auomaon Czech Academy of Scences Pod Vodárenskou věží 1143/ Prague 8 Czech Republc ABSTRACT The sae-of-ar mehods for dsrbued esmaon of mxures assume he exsence of a common mxure model. In many praccal suaons hs assumpon may be oo resrcve as a subse of parameers may be purely local e.g. f he numbers of observable componens dffer across he nework. To reflec hs ssue we propose a new onlne Bayesan mehod for smulaneous esmaon of local parameers and dffuson esmaon of global parameers. The algorhm consss of wo seps. Frs he nodes perform local esmaon from own observaons by means of facorzed pror/poseror dsrbuons. Second a dffuson opmzaon sep s used o merge he nodes global parameers esmaes. A smulaon example demonsraes mproved performance n esmaon of boh parameers ses. Index Terms Dffuson esmaon dsrbued esmaon exponenal famly mxure models message passng. 1. INTRODUCTION Dsrbued esmaon of parameers of sochasc models has araced a sgnfcan aenon n he las wo decades parcularly due o a rapd developmen of cheap ad-hoc wreless sensor neworks conssng of nodes endowed wh sensng daa processng and communcaon capables. Ther applcaons range from localsaon arge rackng nruson deecon dconary learnng ec. [1]. Accordng o he communcaon sraeges among sources several groups of mehods can be recognzed. Frs he ncremenal algorhms passng and processng nformaon n a Hamlonan cyclc pah. These algorhms are prone o falures as each node and lnk are sngle-pons of falure and recovery s an NP-hard problem [2]. Ths ssue can be solved by he dffuson and consensus sraeges. The former perform only a lmed amoun of communcaon among neghborng nodes ypcally represened by an exchange of observaons durng he adapaon sep and an exchange of esmaes durng he combnaon sep [1 3]. The sandard consensus sraeges nvolved nermedae seps o reach consensus bu he recen runnng consensus algorhms removes hs overhead e.g. [4]. In hs paper we specfcally focus on sequenal esmaon of mxure models. Under roughly common dsrbuon of observaons y across he nework here exs several mehods for collaborave esmaon of mxures. One branch of soluons s based on expecaon-maxmzaon EM algorhm. For nsance Gu proposed a verson of a decenralzed EM algorhm wh a consensus sep evaluang global nework sascs beween he sandard local E K. Dedecus s suppored by he Czech Scence foundaon gran No P. V. Sečkárová s suppored by he Czech Scence Foundaon gran No S. and M seps [5]. A smlar approach called dsrbued expecaonmaxmzaon was proposed by Safarnejadan Menha and Karrar [6] where beween he E and M seps a dsrbued averagng approach s used o dffuse local suffcen sascs o neghborng nodes and esmae global suffcen sascs n each node. In he M-sep each node updaes parameers of he normal mxure model usng he esmaed global suffcen sascs. A fully dffuson-orened EM algorhm was proposed by Perera Pages- Zamora and opez-valcarce n [7]. Anoher branch of soluons semmng from he Bayesan heory s mosly based on varaonal nference. The dsrbued varaonal Bayesan algorhm DVBA of Safarnejadan Menhaj and Karrar allows dsrbued esmaon of normal mxure models [8]. However DVBA s an ncremenal algorhm. In order o remove he robusness ssue of hs opology he auhors laer proposed a peer-o-peer DVBA algorhm where he nodes communcae wh her neghbors [9]. Recenly Dedecus Rechl and Djurć proposed a dsrbued quas-bayesan algorhm wh pon-esmaed componen ndcaor applcable o a wde class of mxure models wh componen dsrbuons from he exponenal famly and suable for onlne esmaon [10]. Wh he excepon of componen weghs n [5] he lsed algorhms rely on he assumpon of a leas roughly common model parameers. However he parameers may dffer across he nework e.g. f he nodes observe dfferen arges maneuverng n a formaon. Then he collaboraon may be based on an assumpon of correlaon or smlary of nferred parameers [11]. These mulask dffuson algorhms are mosly MS-based e.g. [11 12] wh a few modfcaons reflecng for nsance sparsy [13]. Some more general recen algorhms allow unsupervsed deermnaon of neghbors belongng o he same cluser [ ]. These algorhms allow sac or sequenal collaborave esmaon of local and global parameers and parameers shared by clusers of nodes. We would lke o conrbue o hs opc from he probablscally conssen and versale Bayesan vewpon allowng absrac formulaon of nference asks. In parcular we propose a new mehod for collaborave esmaon of mxure models wh local and global parameer ses. The confguraon of parameers s very general he mxures may dffer n he number of componens her weghs and he componens may have dfferen parameers oo. The mehod s generc and applcable o a wde class of mxure models. Under ceran condons smplfes o an analycally racable mean-feld varaonal mehod. The resulng algorhm consss of wo phases. Frs he local esmaon where he nodes perform nference of boh local and global parameers ses locally from own observaons. Second he dffuson opmzaon sep where he nodes exchange he dsrbuons of global parameers wh her neghbors whn 1 hop dsance. If he mxure componens are exponenal 362

2 famly dsrbuons he whole algorhm can be reduced o varaonal message passng e.g. [17] exended by message-passng dffuson opmzaon. 2. PROBEM STATEMENT e us consder a nework a dreced or undreced graph of nodes I = {1... I} conneced by a se of edges deermnng he graph opology. Each node I acqures observaons y where = s a dscree me ndex. These observaons can be modelled by mxure models wh a common global subse of parameers. Each node also communcaes wh s adjacen neghbors whn 1 hop dsance formng s neghborhood I. We emphasze ha I oo. The adoped communcaon sraegy s dffuson [1 3] however compared o he ordnary dffuson algorhms he adapaon sep for an observaons exchange among nodes s no applcable here for her poenally dfferen dsrbuons. The combnaon sep mergng he parameers esmaes s preserved n a specal form. The ncomng observaons y of node follow a mxure dsrbuon.e. a convex combnaon of K probably dsrbuons called componens. Ther probably denses are denoed p k y θ k where θ k are componen parameers and k = 1... K are componen ndces. Denong by Θ all unknown parameers of he consdered mxure we have K p y Θ = φ k p k y θ k 1 k=1 where φ = [φ 1... φ K] s a vecor of componen weghs ha s posve real numbers akng values n he K 1-dmensonal probablsc smplex. Noe ha Θ may be any combnaon of φ θ or K. As a mnor smplfcaon we assume he local numbers of componens K o be known a pror. We face he problem of sequenal collaborave esmaon of nodes mxures parameers wh he assumpon ha here exss a global subse of parameers Θ G ha s common o all nework nodes and a local subse ha s s local complemen o Θ Θ G = I Θ and Θ = Θ \ Θ G. The goal s o explo he exsng nework n order o collaboravely arrve a beer esmaes of Θ G under modes communcaon requremens. In addon we conjecure ha an mprovemen n he esmaon of Θ G wll nduce an mproved esmaon of Θ. 3. SEQUENTIA DIFFUSION ESTIMATION The ordnary sequenal Bayesan nference of unknown parameers Θ reles on a jon pror dsrbuon π Θ y 0: 1 = π Θ Θ G y0: 1 quanfyng he curren sascal knowledge abou Θ and Θ G up o me 1 whch s based on he prevous observaons y 1... y 1 and he pseudo-observaons y 0 deermnng any nal knowledge e.g. from hsorcal daa or an exper s opnon. The ncorporaon of new observaon y s performed by vrue of he Bayes heorem π Θ y 0: = π Θ y 0: 1p y Θ πθ y 0: 1p y Θ dθ. 2 Unforunaely hs rgorous Bayesan approach o nference of Θ s mpraccal no only compuaonally bu also from aspec of he dsrbued esmaon. In order o combne nformaon from several sources nodes he dsrbuon of Θ G mus have he same form across he nework condonally ndependen of any local parameers Θ. Tha means π Θ y 0: = π Θ Θ G y0: ρ Θ y0: ρ Θ G y0:. 3 In oher words he rue dsrbuon of Θ and Θ G a each node I mus be replaced by wo lower-dmensonal dsrbuons ρ Θ and ρ Θ G he laer wh he same funconal form for all and whose produc s as close o he orgnal dsrbuon as possble n he Kullback-ebler sense. Somemes hs facorzaon s naural e.g. n mxure esmaon where he componen weghs may be modelled as ndependen on componen parameers. More on possble facorzaons can be found n [18]. The second sep s he dffuson opmzaon of he dsrbuon of Θ G. From node I vewpon s possble o vew he neghbors dsrbuons ρ jθ G as hypoheses abou Θ G. The goal s o opmally merge hese hypoheses no a sngle dsrbuon ρ Θ G ha s as close o ndvdual hypoheses as possble. Conssenly wh he Bayesan heory boh seps wll explo he Kullback-ebler dvergence as he measure of proxmy of probably dsrbuons. For wo probably denses fx and gx he laer absoluely connuous wh respec o he former he Kullback- ebler dvergence s defned as [ D [fx gx] = E fx log fx ]. 4 gx Ths dvergence s a premerc: s nonnegave and equal o zero f f = g almos everywhere bu s neher symmerc nor does sasfy he rangle nequaly. For he sake of smplcy we adop he assumpon ha he nodes know whch parameers are local and whch are global. Ths suaon occurs e.g. f dfferen ypes of sensors are used for akng observaons. The proposed algorhms s as follows: Algorhm formulaon We am o desgn a dsrbued algorhm for onlne mxures esmaon performng he followng wo seps: 1. ocal esmaon: A each node I locally esmae parameers Θ and Θ G n a Kullback-ebler-opmal facorzed form D [ρ Θ ρ Θ G π Θ Θ G ] mn. 2. Dffuson opmzaon: A each node I fnd he Kullback-ebler-opmally merged dsrbuon ρθ G usng he neghbors dsrbuon ρ jθ G as hypoheses abou Θ G ρ Θ G = arg mn I 1 D ρ [ ρ Θ G ρj Θ G ]

3 3.1. ocal esmaon ocal esmaon assers approxmaon of he nracable rue densy π Θ Θ G by ndvdual lower-dmensonal denses ρ Θ and ρ Θ G as n 5. Tha s he goal s o perform mnmzaon of he Kullback-ebler dvergence D [ρ Θ Θ G π Θ Θ G ] y 0: [ ρ Θ Θ G ] = E ρ Θ ΘG log π Θ Θ G y0: [ ] = E ρ Θ log ρ Θ + E ρ Θ G [log ρ Θ G ] ] E ρ Θ E ρ Θ G [log π Θ Θ G. 6 I s sraghforward o see ha he mnmzaon of 6 wh respec o Θ and Θ G leads o mnmzaons of expeced Kullback-ebler dvergences wh lower-dmensonal denses. Because he mnmum s reached under equal argumens of he Kullback-ebler dvergence he soluon s gven by he sysem ρ Θ G ]} = c exp {E ρθ [log π Θ Θ G { ]} ρ Θ = c G exp E ρθ G [log π Θ Θ G where c and c G are normalzaon consans. The presened facorzaon s closely relaed o he mean-feld varaonal Bayesan nference [19 20]. Indeed s possble o furher facorze he parcular denses ρ Θ and/or ρ Θ G and proceed wh he ordnary varaonal Bayes mehod sequenally seekng a conssen soluon by revsng he lower-dmensonal denses n a crcular way unl a convergence creron s me. The mean-feld varaonal algorhms are guaraneed o converge [21]. In he sequenal esmaon framework usually a few eraons are performed a each me sep usng prevous observaons sored n memory. The poseror dsrbuons from he prevous me sep hen serve as he pror dsrbuons. More elaboraed mehods for onlne varaonal nference suable for large daa can be found n [22 23]. In he nex secon he descrbed local esmaon va facorzed denses wll be exended by dffuson opmzaon of he poseror dsrbuon of Θ G. Then wll be shown ha f he dsrbuons of he mxure componens belong o he exponenal famly he proposed mehod s analycally racable as a message passng algorhm Dffuson opmzaon Each node I has access o denses ρ jθ G of s neghbors j I represenng hypoheses abou he rue parameer Θ G. Insead of workng wh he whole sysem of ndvdual denses we wan o replace hem by a sngle densy ρ Θ G. In parcular we seek an elemen from he se of all admssble denses mnmzng he average dvergence o all ndvdual denses ρ Θ G = arg mn I [ 1 D ρ ρ Θ G ρ jθ G ] = arg mn D ρ ρ Θ G = [ ρ jθ G ] 1/ I [ρ j Θ G ] 1/ I. 7 The resulng Kullback-ebler-opmal dsrbuon ρ Θ G s hus a geomerc average of neghbors dsrbuons. In he Bayesan dffuson framework hs mergng concdes wh he combne sep preferably performed on neghbors poseror dsrbuons [25]. However n our case he local esmaon mehod s erave and 7 may be used beween any subsequen eraons o speed up convergence. Ths promsng opc s posponed o furher research. Generally he varaonal esmaon of he poseror dsrbuon s naccurae n he early sage of he onlne learnng and gradually becomes accurae as learnng proceeds [24]. Therefore s reasonable o sar wh a hgher number of local eraons o speed up hs convergence and o decrease laer down o one eraon beween wo dffuson seps. The proposed collaboraon by fuson of poseror dsrbuons can be seen as a weghed Bayesan learnng ha conrbues o he nference process hs wll become more apparen n he ongong secon. 4. COMPONENTS FROM THE EXPONENTIA FAMIY If he mxure componens belong o he exponenal famly of dsrbuons he local esmaon can ake he form of a message passng algorhm. In general any random varable y has an exponenal famly dsrbuon wh a parameer ϑ f s probably densy funcon can be wren n he form py ϑ = exp [η T yy Aη ky ] 8 where η ηϑ s a naural parameer T yy s a suffcen sasc Aη s a log-paron normalzng funcon and ky s a funcon of y. The parcular form s no unque. The Bayesan esmaon of ϑ s analycally racable f he pror dsrbuon of ϑ s conjugae.e. parameerzed by hyperparameers ξ 1 of he same sze as T yy and real scalar ν 1 and wh a densy of he form π ϑ ϑ ξ 1 ν 1 = exp [η ξ 1 ν 1Aη] exp [hξ 1 ν 1] 9 where Aη s he same funcon as n he exponenal famly dsrbuon and hξ 1 ν 1 s a known funcon. The poseror dsrbuon of ϑ s hen gven by updaed hyperparameers ξ = ξ 1 + T yy and ν = ν In he local esmaon sep he componen denses p k y θ k from Equaon 1 and he convenen pror dsrbuons for θ k are rewren o compable forms accordng o 8 and 9. Smlarly he componen ndcaors for y deermnng whch componen generaed he acual y are modelled by mulnomal dsrbuons wh he probables.e. componen weghs φ k provded by he conjugae Drchle dsrbuon. The local esmaon algorhm hen eraes by passng messages expecaons from he pror dsrbuons o he componens and mulnomal ndcaor models whch n urn reurn messages conanng he suffcen sascs updang he pror hyperparameers n a sense 10. Ths s known as he varaonal message passng [17]. From 9 and he dffuson opmzaon sep 7 follows ha mergng of poseror dsrbuons akes he form ξ = 1 ξ j and ν = 1 ν j. I I 364

4 Remnd ha can be performed beween any wo subsequen eraons of he message passng algorhm e.g. o speed up convergence or afer he local esmaon sep o save communcaon resources. Also remark s prncpal smlary wh he Bayesan updae under conjugacy 10 hs s where he cornersone of hs mergng les. 5. SIMUATION EXAMPE Ths example demonsraes he sequenal dffuson esmaon of a normal mxure model by a nework conssng of 16 nodes. Is opology s depced n Fgure 1. The mxure has he form K y Θ Θ G φ k N µ k Σ k k=1 where µ k and Σ k are he mean vecors and covarance marces. The frs componen observed by all nework nodes has parameers [ ] [ ] ε 0.9 µ 1 = Σ 5 1 = ε where ε ε Exp0.5 are..d. random samples from he exponenal dsrbuon. The second componen s observed by nodes { } only. Ther parameers are [ [ ] ε 0.8 µ 2 = Σ 0] 2 = ε where ε ε Exp0.5 are..d. random samples from he exponenal dsrbuon. The componen weghs n nodes { } are φ = [ ]. The parameers ses are Θ G = {µ 1} and Θ = {µ 2 Σ 1 Σ 2 π }. Noe ha here s a poenal for cooperaon n esmaon of µ 2 and φ. The esmaon sars from he nal normal N nverse- Wshar W and Drchle Dr pror dsrbuons µ 1 N [ 7 7] 15 I [2 2] µ 2 N [7 7] 15 I [2 2] Σ 1 Σ 2 W I [2 2] π Dr[1 1]. The sngle-componen nodes provde her nformaon o all neghbors bu ncorporae ρ Θ G only from oher sngle-componen nodes observaons were generaed for each node he esmaon sars when he frs 100 observaons are receved. A each me 5 eraons of he local esmaon sep are performed hen he dffuson opmzaon follows. As a performance measure we employ MSE averaged over he nework denoed by AMSE. The evoluon of AMSEs under cooperaon and no-cooperaon are depced n Fg. 2 for µ 1 and µ 2 Fg. 3 for Σ 1 and Σ 2 and Fg. 4 for π. We conclude ha he proposed mehod leads o a sgnfcan mprovemen n esmaon of µ 1. Moreover smulaneously helps he esmaon of oher parameers. Snce he memory lengh for y would be lmed n pracce we also performed a smulaon wh a queue-ype memory for 100 observaons. The resuls were very smlar o hose presened here he mehod performed very slghly worse n erms of AMSE. Fnally we remark ha [5] provdes an algorhm ha allows dsrbued mxure esmaon wh nhomogeneous componen weghs across he nework bu assumes dencal componens Fg. 1. Topology of he dffuson nework Coop. No coop. µ 1 µ Fg. 2. Evoluon of logarhm of AMSE of µ 1 and µ 2 under cooperaon Coop. and no cooperaon No coop Coop. No coop. 0.5 Fg. 3. Evoluon of logarhm of AMSE of Σ 1 and Σ 2 under cooperaon Coop. and no cooperaon No coop Σ Σ2 Coop. No coop. Fg. 4. Evoluon of logarhm of AMSE of π under cooperaon Coop. and no cooperaon No coop.. 365

5 6. REFERENCES [1] A. H. Sayed Adapve neworks Proceedngs of he IEEE vol. 102 no. 4 pp Apr [2] C. H. Papadmrou Compuaonal Complexy Addson Wesley [3] A. H Sayed Adapaon learnng and opmzaon over neworks Foundaons and Trends n Machne earnng vol. 7 no. 4-5 pp [4] P. Braca S. Marano and V. Maa Runnng consensus n wreless sensor neworks n 11h Inernaonal Conference on Informaon Fuson June 2008 pp [5] D. Gu Dsrbued EM Algorhm for Gaussan Mxures n Sensor Neworks IEEE Transacons on Neural Neworks vol. 19 no. 7 pp July [6] B. Safarnejadan M. B. Menhaj and M. Karrar Dsrbued unsupervsed Gaussan mxure learnng for densy esmaon n sensor neworks IEEE Transacons on Insrumenaon and Measuremen vol. 59 no. 9 pp Sep [7] S. S. Perera A. Pages-Zamora and R. opez-valcarce A dffuson-based dsrbued EM algorhm for densy esmaon n wreless sensor neworks n 2013 IEEE Inernaonal Conference on Acouscs Speech and Sgnal Processng. May 2013 pp [8] B. Safarnejadan M.B. Menhaj and M. Karrar Dsrbued varaonal Bayesan algorhms for Gaussan mxures n sensor neworks Sgnal Processng vol. 90 no. 4 pp Apr [9] B. Safarnejadan and M. B. Menhaj Dsrbued densy esmaon n sensor neworks based on varaonal approxmaons Inernaonal Journal of Sysems Scence Apr [10] K. Dedecus J. Rechl and P.M. Djurć Sequenal esmaon of mxures n dffuson neworks IEEE Sgnal Processng eers vol. 22 no. 2 pp [11] J. Chen C. Rchard and A.H. Sayed Dffuson MS over mulask neworks IEEE Transacons on Sgnal Processng vol. 63 no. 11 pp [12] J. Plaa-Chaves N. Bogdanovć and K. Berberds Dsrbued dffuson-based MS for node-specfc adapve parameer esmaon IEEE Transacons on Sgnal Processng vol. 63 no. 13 pp July [13] R. Nassf C. Rchard A. Ferrar and A.H. Sayed. Mulask dffuson MS wh sparsy-based regularzaon. Proc. IEEE Inernaonal Conference on Acouscs Speech and Sgnal Processng ICASSP [14] J. Chen C. Rchard A. O. Hero and A. H. Sayed Dffuson MS for mulask problems wh overlappng hypohess subspaces n 2014 IEEE Inernaonal Workshop on Machne earnng for Sgnal Processng MSP Sep pp. 1 6 IEEE. [15] X. Zhao and A. H. Sayed Dsrbued cluserng and learnng over neworks IEEE Transacons on Sgnal Processng vol. 63 no. 13 pp July [16] S. Khawam A.M. Zoubr and A.H. Sayed. Decenralzed cluserng over adapve neworks. Proc European Sgnal Processng Conference [17] J. Wnn and C.M. Bshop Varaonal message passng Journal of Machne earnng Research vol. 6 pp [18] C.M. Bshop Paern Recognon and Machne earnng Sprnger [19] M. I. Jordan Z. Ghahraman T. S. Jaakkola and. K. Saul An nroducon o varaonal mehods for graphcal models Machne earnng vol. 37 no. 2 pp [20] T. S. Jaakkola Tuoral on varaonal approxmaon mehods n Advanced Mean Feld Mehods: Theory and Pracce 2000 pp [21] S. Boyd and. Vandenberghe Convex Opmzaon Cambrdge Unversy Press [22] M. Hoffman F. R. Bach and D. M. Ble Onlne learnng for laen Drchle allocaon n Advances n Neural Informaon Processng Sysems 23 pp Curran Assocaes Inc [23] T. Broderck N. Boyd A. Wbsono A. C. Wlson and M.I. Jordan Sreamng varaonal Bayes n Advances n Neural Informaon Processng Sysems 26 pp Curran Assocaes Inc [24] M. Sao Onlne model selecon based on he varaonal Bayes Neural Compuaon vol. 13 no. 7 pp [25] K. Dedecus and V. Sečkárová Dynamc dffuson esmaon n exponenal famly models IEEE Sgnal Processng eers vol. 20 no. 11 pp Nov [26] D. A. Knowles and T. Mnka Non-conjugae varaonal message passng for mulnomal and bnary regresson n Advances n Neural Informaon Processng Sysems 24 pp Curran Assocaes Inc [27] M. P. Wand Fully smplfed mulvarae normal updaes n non-conjugae varaonal message passng Journal of Machne earnng Research vol. 15 pp