Generative Models and Naïve Bayes

Transcription

1 COM4 Mhe Lerg Geertve Models d Nïve Byes Ke Che Redg: [4.3 EA] [3.5 KM] [.5.4 CMB]

2 Outle Bkgroud d robblty Bss robblst Clssfto rple robblst dsrmtve models Geertve models d ther pplto to lssfto MA d overtg geertve to dsrmtve Nïve Byes geertve model rple d Algorthms dsrete vs. otuous Emple: ly Tes Zero Codtol robblty d Tretmet Summry COM4 Mhe Lerg

3 Bkgroud There re three methodologes: Model lssfto rule dretly Emples: k-nn ler lssfer SVM eurl ets b Model the probblty of lss membershps gve put dt Emples: logst regresso probblst eurl ets softm Mke probblst model of dt wth eh lss Emples: ve Byes model-bsed. Importt ML toomy for lerg models probblst models vs o-probblst models dsrmtve models vs geertve models COM4 Mhe Lerg 3

4 Bkgroud Bsed o the toomy we see the essee of dfferet supervsed lerg models lssfers more lerly. robblst No-robblst Dsrmtve Logst Regresso robblst eurl ets.. K- Ler lssfer SVM Neurl etworks Geertve Nïve Byes Model-bsed e.g. GMM N.A.? COM4 Mhe Lerg 4

5 robblty Bss ror odtol d jot probblty for rdom vrbles ror probblty: Codtol probblty: Jot probblty: Reltoshp: Idepedee: Byes Rule Dsrmtve Geertve COM4 Mhe Lerg Lkelhood ror osteror Evdee 5

6 robblst Clssfto rple Estblshg probblst model for lssfto Dsrmtve model L To tr dsrmtve lssfer regrdless ts probblst or o-probblst ture ll trg emples of dfferet lsses must Dsrmtve be jotly used to buld up sgle robblst Clssfer dsrmtve lssfer. Output L probbltes for L lss lbels probblst lssfer whle sgle lbel s heved by o-probblst dsrmtve lssfer. L COM4 Mhe Lerg 6

7 robblst Clssfto rple Estblshg probblst model for lssfto ot. Geertve model must be probblst L Geertve robblst Model for Clss Geertve robblst Model for Clss L L L probblst models hve to be tred depedetly Eh s tred o oly the emples of the sme lbel Output L probbltes for gve put wth L models Geertve mes tht suh model produe dt subjet to the dstrbuto v smplg. COM4 Mhe Lerg 7

8 robblst Clssfto rple Mmum A osteror MA lssfto rule For put fd the lrgest oe from L probbltes output by dsrmtve probblst lssfer.... Assg to lbel * f * s the lrgest. Geertve lssfto wth the MA rule Apply Byes rule to overt them to posteror probbltes for L The pply the MA rule to ssg lbel L Commo ftor for ll L probbltes COM4 Mhe Lerg 8

9 COM4 Mhe Lerg 9 Nïve Byes Byes lssfto Dffulty: lerg the jot probblty s ofte fesble! Nïve Byes lssfto Assume ll put fetures re lss odtolly depedet! Apply the MA lssfto rule: ssg to * f.... for L L ] [ ] [ * * * * > ' Applyg the depedee ssumpto estmte of * of estmte

10 Nïve Byes For eh trget vlue of ˆ estmte For every feture vlue ˆ j jk wth emples S; jk estmte of eh feture jk L j j F; k wth emples S; N j ' ˆ ˆ ˆ ˆ * * * * [ ] > [ ] ˆ ˆ L COM4 Mhe Lerg 0

11 Emple: ly Tes Emple COM4 Mhe Lerg

12 Emple Lerg hse Outlook lyyes lyno Suy /9 3/5 Overst 4/9 0/5 R 3/9 /5 Temperture lyyes lyno Hot /9 /5 Mld 4/9 /5 Cool 3/9 /5 Humdty lyyes lyno Wd lyyes lyno Hgh 3/9 4/5 Norml 6/9 /5 Strog 3/9 3/5 Wek 6/9 /5 lyyes 9/4 lyno 5/4 COM4 Mhe Lerg

13 Test hse Emple Gve ew ste predt ts lbel OutlookSuy TempertureCool HumdtyHgh WdStrog Look up tbles heved the lerg phrse OutlookSuylyYes /9 TempertureCoollyYes 3/9 HumtyHghlyYes 3/9 WdStroglyYes 3/9 lyyes 9/4 Deso mkg wth the MA rule OutlookSuylyNo 3/5 TempertureCoollyNo /5 HumtyHghlyNo 4/5 WdStroglyNo 3/5 lyno 5/4 Yes [SuyYesCoolYesHghYesStrogYes]lyYes No [SuyNo CoolNoHghNoStrogNo]lyNo Gve the ft Yes < No we lbel to be No. COM4 Mhe Lerg 3

14 Nïve Byes Algorthm: Cotuous-vlued Fetures Numberless vlues tke by otuous-vlued feture Codtol probblty s ofte modelled wth the orml dstrbuto µ σ j j j µ j ˆ j ep πσ j σ j : me verge of feture vlues of emples : stdrd devto of feture vlues Lerg hse: F L Output: F L orml dstrbutos d C L Test hse: Gve ukow ste X Isted of lookg-up tbles lulte odtol probbltes wth ll the orml dstrbutos heved the lerg phrse Apply the MA rule to ssg lbel the sme s doe for the dsrete se COM4 Mhe Lerg j j of emples for whh for X X X C for whh 4

15 Nïve Byes Emple: Cotuous-vlued Fetures Temperture s turlly of otuous vlue. Yes: No: Estmte me d vre for eh lss N N µ Yes.64 µ σ µ N µ No 3.88 N σ σ Yes No Lerg hse: output two Guss models for tempc ˆ Yes.35 ˆ No 7.09 ep π ep π COM4 Mhe Lerg ep π ep π

16 Zero odtol probblty If o emple ots the feture vlue I ths rumste we fe zero odtol probblty problem durg test ˆ ˆ ˆ 0 for ˆ 0 jk jk jk For remedy lss odtol probbltes re-estmted wth : umber of + mp ˆ jk + m trg emples for whh : umber of trg emples for whh p : pror estmte usully p / t for t possble vlues of m : weght to pror umber of j m-estmte "vrtul"emples j jk d m j COM4 Mhe Lerg 6

17 Zero odtol probblty Emple: outlookoversto0 the ply-tes dtset Addg m vrtul emples m: tuble but up to % of #trg emples I ths dtset # of trg emples for the o lss s 5. Assume tht we dd m vrtul emple our m-estmte tretmet. The outlook feture tkes oly 3 vlues. So p/3. Re-estmte outlooko wth the m-estmte COM4 Mhe Lerg 7

18 Summry robblst Clssfto rple Dsrmtve vs. Geertve models: lerg vs. Geertve models for lssfto: MA d Byes rule Nïve Byes: the odtol depedee ssumpto Trg d test re very effet. Two dfferet dt types led to two dfferet lerg lgorthms. Nïve Byes: populr geertve model for lssfto erforme ompettve to my stte-of-the-rt lssfers eve the presee of voltg the odtol depedee ssumpto My suessful ppltos e.g. spm ml flterg A good ddte of bse lerer esemble lerg COM4 Mhe Lerg 8