Pessimistic decision tree pruning. based on tree size. Yishay Mansour. Computer Science Dept. Tel-Aviv University.

Transcription

1 Pessimistic decisio tee puig based o tee size Yishay Masou Compute Sciece Dept. Tel-Aviv Uivesity Tel-Aviv, ISRAEL masou@math.tau.ac.il Abstact I this wok we develop a ew citeia to pefom pessimistic decisio tee puig. Ou method is theoetically soud ad is based o theoetical cocepts such as uifom covegece ad the Vapik-Chevoekis dimesio. We show that ou citeia is vey well motivated, fom the theoy side, ad pefoms vey well i pactice. The accuacy of the ew citeia is compaable to that of the cuet method used i C Itoductio The pheomea of ovettig the data is well kow i machie leaig, ad efes to the case that the leaed hypothesis is so closely elated to the taiig examples such that its geealizatio capabilities would be pealized. Ovettig would usually occu whe the class of hypotheses used is as complex as the give taiig sample. Fo this easo we would like, i may cases, to limit the hypothesis we geeate to be \less complex" tha the taiig sample. I decisio tees the ovettig pheomea ca occu whe the size of the tee is too lage compaed to the umbe of taiig examples. As a exteme case, fo a give taiig sample coside a abitay decisio tee whee each leaf cotais at most a sigle example fom the taiig sample. By settig the value of the leaf to be the label of the example that eaches it, we ae guaatee to have o eo o the taiig sample. Howeve, the geealizatio capabilities of such a tee ae vey doubtful. Fo this easo we ae iteested ot oly i tees whose eo o the taiig sample is small, but also those whose size is small compae to the sample size. Thee ae two geeal methods, i decisio tee iductio, to boud the size of the tee. The st is to have some citeia, called Ealy Stoppig citeia, that would be used i ode to decide whethe o ot to stop gowig a give ode. Such a ealy stoppig citeia limits the size of the tee ad avoids tyig to gow uecessay sub-tees. The isk i usig a ealy stoppig citeia is that by ot gowig a sub-tee we might be missig some eal stuctue of the poblem that could be discoveed oly late. The secod method is gowig the iitial tee util it ts the sample pefectly. The we use this tee as ou iitial statig poit ad stat puig it. Geeally, puig out pefoms ealy stoppig, sice it allows us to discove some stuctue that is ot immediate. Thee ae a few dieet methods of pefomig the puig ad they all output at the ed of the puig pocess a tee which is a pued vesio of the oigial tee. Thee ae two dieet methodologies fo pefomig puig. The st is of geeatig a sequece of pued tees, ad late to chose oe of them. This idea is the coe idea behid the Miimal Cost-Complexity Puig poposed by Beima et. al. [BFOS84]. They poposed usig dyamic pogammig to d a sequece of tees that miimizes a combiatio of the eo o the taiig sample ad the tee size. Late, a idepedet sample is used to chose the best tee out of the sequece. I [BB94] a simple dyamic pogammig algoithm is give to d the smallest pued decisio tee that has a give eo ate. Hee agai, oe ca geeate a sequece of tees ad use a idepedet sample to chose the best tee i the sequece. The secod methodology fo puig ties to diectly pue the oigial decisio tee. The puig is doe by eplacig the sub-tee ooted at a ode by eithe a leaf o by a sub-tee ooted at oe of its childe.

2 The appoach of Reduced Eo Puig, poposed by Quila [Qui87], uses a idepedet sample to test the accuacy of each sub-tee compaed to the case whe it is pued. Miges [Mi87] suggested the Citical Value Puig, which pefoms the decisio about puig usig o the \ifomatio-gai" that was achieved i gowig the tee. (Fo each ode it computes the maximum \ifomatio-gai" i its subtee, ad pues a ode if this value is less tha a cetai theshold.) Ude the categoy of pessimistic puig we iclude all the puig citeias that ty to boud the tue eo of a sub-tee based o its taiig eo. We pefom the puig at a ode whe the eductio i the eo due to the sub-tee seems statistically isigicat. The algoithmic beet of the pessimistic puig methods is that they equie oly a sigle pass ove the tee to geeate the pued tee (compaed to the quadatic time of the dyamic pogammig based methods). The mai questio ow is how do we decide if the chage i the eo is sigicat. We ty to estimate (o at least to uppe boud) the tue eo of a subtee, based o the umbe of examples it classies, the umbe of eos, the umbe of classes, ad the size of the tee. Oe method that falls ito ou categoy of pessimistic puig is Miimum eo puig [Nib87]. This method ties to estimate the tue eo usig the followig fomula, e + (? 1) + whee is the umbe of classes, the umbe of examples, ad e is the umbe of eos. (The theoetical justicatio fo the fomula is simple. Assumig a uifom pio ove the classes, the above is the expected eo ove the posteio distibutio, give that i examples thee whee e eos.) Pessimistic puig of Quila [Qui86, Qui93] uses a complex method based o the umbe of eos ad the size of the taiig sample. Howeve, the theoetical justicatio of this method is athe weak, o as Quila says i his book \: : : the easoig should be take with a lage gai of salt" [Qui93]. This method of pessimistic puig is the method that is used cuetly i C4.5. A excellet suvey ad empiical compaiso of the dieet puig methods was doe by Miges [Mi89], which also icludes a suvey of all the dieet methods. Ou statig poit ca be motivated as follows. Assume you have a sub-tee that classies 200 taiig examples ad makes o eo. We would vey much like to dieetiate the case that the sub-tee is of a small size (say 3) o a huge size (say 100). I the fome case we expect that the tue eo would be vey small, while i the latte case the tue eo might be huge. This is a most compellig agumet why the size of the decisio tee should be icopoated i the pessimistic estimate. Ou basic poposal it to estimate the eo of a sub-tee o the taiig sample by the followig quatity, e + c k log d + c 2 1 ; whee is the sample size that eaches the sub-tee, e is the umbe of icoect classicatios doe by the sub-tee, k is the size of the sub-tee, d the umbe of attibutes ad c 1 ad c 2 ae some costats. We show, usig basic theoetical agumets, that the above expessio is a uppe boud o the tue eo of the sub-tee. Whe cosideig pessimistic puig oe should questio how \pessimistic" is it. The mai poit to keep i mid is that the geeated decisio tee is ot abitay, but athe the oe that educes the eo o the taiig examples. Fo example, say we have a set of fuctios all with the same tue eo. O ay give set of examples, we expect to have some fuctio, fom the set of fuctios, whose eo o this set of examples is lowe tha the tue eo. This is why ou pessimistic estimates ae eally ealistic estimates i may cases. We have pefomed empiical tests to compae the cuet method of C4.5 ad ou ew poposed method. We stat with a aticial data o which we show that ou citeia is supeio to the cuet method of C4.5 (eve whe modifyig C4:5's codece paamete). We the cotiue ad pefom a empiical evaluatio of six datasets. Sice ou theoy was desig fo boolea output, we cocetated o datasets whee the output is boolea. The compaiso shows that the accuacy of the ew citeia is compaable to that used i C4.5. (Howeve, it ejoys beig theoetically soud.) We show that i ou method we ca use the paamete c to cotol the size of the pued tees. A lage value of c will cause moe puig, while a smalle value would limit the puig. (This is simila to the codece paamete build i C4.5, which also cotols the amout of puig.)

3 Fom ou empiical esults it seems that vey sigicat puig geeates much smalle decisio tees ad maitais the accuacy. I [AHM95] the accuacy of C4.5 was compaed to a algoithm that geeates decisio tees of depth two. Thee it was obseved that decisio tees of depth two achieve emakably good esults compaed to C4.5, although i some cases the limitatio to depth two foced educed accuacy. I this wok we obseved that ou citeia ca be used to pefom exteme puig of the decisio tee while maitaiig its accuacy. This might lead oe to suspect that the complex decisio tees, that ae i may cases geeated by the default settig of C4.5, have thei pedictive powe cocetated i a small sub-tee. We may hope that by exteme puig we will ecove that sub-tee. This wok is ogaized as follows. I Sectio 2 we deive the theoetical bouds. Sectio 3 descibes ou citeia i detail. I Sectio 4 we descibe the empiical esults. 2 Theoetical Backgoud I this sectio we deive fom basics the bouds that justify ou citeia. Moe elaboate techiques may give supeio bouds, but we did ot ty to peuse that diectio. We ae iteested i the maximum deviatio of the eo whe the hypothesis is a biay decisio tee with k odes. Give this we will be able to boud the tue eo by the eo o the sample plus the maximum deviatio. The st techical lemma that we deive bouds the umbe of possible decisio tees. Lemma 2.1 The umbe of biay decisio tees ove d biay iputs with boolea output ad k = 2k 0? 1 odes is bouded by, is O(2 k ) d k0?1 2 k0 = O((4d) k ) = O((4d) k ): Poof: I a biay decisio tee, each iteal ode has degee two (othewise it is edudat). Theefoe, a biay decisio tee with k = 2k 0? 1 odes, has k 0 leaves ad k 0?1 iteal odes. Each leaf ca be label by eithe zeo o oe, theefoe the umbe of assigmets to the leaves is at most 2 k0. Each iteal ode ca be labeled by ay of the iputs, theefoe thee ae at most d k0?1 ways to label the iteal odes. The umbe of dieet biay tees ca by bouded by O(2 k ) (poof omitted). Multiplyig the thee quatities deives the lemma. 2 The ext step is to boud the deviatio of a xed fuctio fom its expectatio. This ca be deived by usig the Cheo iequality. Lemma 2.2 (Cheo) Let x 1 ; : : : ; x be idepedet idetically distibuted adom vaiables, whee x i 2 f0; 1g ad P ob[x i = 1] = p. Let ^p = P i=1 x i=, the: P ob[j^p? pj ] 2e?22 By combiig the two we ca deive the followig theoem, Theoem 2.3 Let H be the class of biay decisio tees ove d biay iputs with boolea output ad at most k odes. Let f be ay boolea fuctio ad D be ay pobability distibutio. Fo h 2 H let e(h; f) = P ob xd [h(x) 6= f(x)]. Give z 1 ; : : : ; z let ^e(h; f) be the factio of z i 's fo which f(z i ) 6= h(z i ). P ob z1;:::;zd fo some costat c. [9h 2 H : je(h; f)? ^e(h; f)j k log d + log 1= > c ] What the above theoem is sayig ca be itepeted as follows. Fo ay decisio tee with k odes, whe tested o examples, the deviatio betwee the obseved eo (o q the sample) ad the tue eo is k log d bouded by O( ), whee we take to be a costat. Such a esult, i the theoy temiology, is a uifom covegece esult, sice the eo of all the hypotheses i H covege simultaeously. This meas that if a give decisio tee h with k odes has a eo ^e(h; f) o a set of examples, its tue eo, e(h; f), with high pobability, is bouded by, k log d + log 1= e(h; f) < ^e(h; f) + c We would like to stess that Theoem 2.3 ca be applied to ay ode i the tee, ad ot oly to the oot. This is due to the atue of uifom covegece esults, that deives wost case boud o the eo, ad the boud holds fo ay distibutio. Theefoe, o matte how we chose the path to the ode (eve advesaially), the boud would still hold. This is due to the obsevatio, that o matte how we estict the distibutio (by the path to the ode, i this case), whe we coside oly examples that have this popety, they

4 algoithm database austalia diabetes gema heat adult Ioosphee No-Puig 143:2 12:3 56:2 9:0 370:4 15:0 61:0 7:2 8742:6 207:9 17:8 4:2 c4.5 -c 1 5:2 3:5 10:2 6:2 2:6 2:6 10:0 4:0 41:6 5:8 12:6 1:9 c4.5 -c 10 13:6 8:5 42:2 6:1 47:0 10:4 27:6 5:9 180:8 10:2 17:0 3:2 c4.5 -c 25 39:8 17:0 49:8 9:4 110:4 18:7 43:0 7:2 675:0 133:2 17:8 4:2 c = 0.5 3:0 0:0 3:4 0:6 1:0 0:0 3:8 0:3 22:0 3:2 3:0 0:0 c = 0.2 6:4 5:4 31:0 8:0 47:0 24:8 14:0 6:0 50:4 14:9 6:2 1:0 c = :8 4:6 47:4 11:5 149:0 14:0 35:0 7:6 214:4 55:9 10:2 2:6 c = :8 17:0 51:8 11:0 240:2 8:7 46:8 8:2 768:6 140:7 14:6 2:9 c = :4 16:3 55:4 8:3 344:2 21:0 58:2 5:8 3368:6 95:5 17:8 4:2 Figue 1: The aveage tee size of each of the citeias. ae daw idepedetly ad idetically fom the esticted distibutio. I this sectio we used vey basic techiques to boud the deviatios. Moe ivolved techiques may be used, ad they could be based o the Vapik Chevoekis (VC) dimesio [VC71]. Fo decisio tees we whee ot able to detemie the exact VC dimesio as a fuctio of thei size, but we ca show that it is betwee (k) ad O(k log d), fo a biay decisio tee ove d iputs with at most k odes (poof omitted). Geeally, the smalle the VC dimesio is the smalle the maximum deviatio would be. 3 Ou Citeia Ou citeia builds diectly o the theoy developed i Sectio 2. We would like, give a obseved eo o the taiig data, to deive a \pessimistic" estimate of the tue eo. The \pessimistic" idicates that we would like ou estimate to be a uppe boud of the tue eo. As we see i the pevious sectio, the theoy q suggests that the deviatio ca be bouded by k log d O( ), whee k is the size of the tee, d the umbe of attibutes ad is the umbe of examples that the tee classies. q k log d Ou pessimistic estimate would simply add O( ) to the obseved eo, ad this would be ou pessimistic estimate of the eo. This would guaatee that with high pobability ou estimate is a uppe boud of the tue eo. Oe issue that eeds to be esolve is the costat that hides i the big O otatio. Rathe tha tyig to deive the exact theoetical costat, we kept it as a fee paamete of ou pogam, deoted by c. I ou expeimets we used dieet values fo c. By cotollig c oe ca cotol the size of the esultig tee, the lage c the moe likely we ae to pue. To be moe specic, assume we chose some value fo c. Assume we ae give a sub-tee of size k, that classies examples (each with d attibutes), e of which it classies icoectly. If we pue the sub-tee, ad eplace it by a leaf, the leaf will make ` eos out of the examples. Ou test whethe to pue the sub-tee is the followig: ` e + c k log d + log 20 : (I C4.5 the puig pocess also icludes compaig a sub-tee to the sub-tees of its childe. We pefom a simila test fo that case.) Note that the modicatio i the eo is based oly o the tee size ad the sample size ad ot o the umbe of eos, i cotast to the cuet method of C4.5 that uses e ad but ot k. 4 Empiical Results We have implemeted ou ew citeia fo pessimistic puig i C4.5 ad compaed it with the existig method of C4.5. (We used C4.5 vesio 8.) Fist we did the compaiso o a aticial data set, this expeimet demostates the beet of icopoatig the tee size ito the puig citeia. We chose the followig aticial data. Each examples has 100 biay attibutes. The classicatio is doe as follows. With pobability 0.9 we chose a adom classicatio ad with pobability 0.1 we chose the value of the st attibute. The distibutio ove the iputs is uifom (i.e. each attibute is idepedet, ad has equal pobability of beig zeo o oe). Clealy, the best classie is simply to use the st attibute, ad it has eo We chose adom examples, ad a C4.5 with

5 algoithm database austalia diabetes gema heat adult Ioosphee No-Puig 20:3 7:2? 27:0 5:0 31:8 3:0 22:6 4:1 15:2 0:2? 14:0 7:2 c4.5 -c 1 16:4 4:9y 24:6 5:6 30:6 1:7 22:6 4:1 14:4 0:2? y 12:8 7:4 c4.5 -c 10 15:9 5:5y 27:5 4:0 29:2 2:4 23:7 4:1 13:9 0:2y 14:2 7:4 c4.5 -c 25 17:1 5:7y 26:4 5:4 30:1 2:1 21:1 3:9 13:9 0:2y 14:0 7:2 c=0.5 15:8 5:6y 27:5 3:0 30:0 2:4 26:7 4:9? y 15:1 0:1? 20:5 9:2? y c=0.2 15:9 5:5y 25:7 4:4 28:5 1:2y 23:0 5:8 14:5 0:2? y 14:8 7:9 c= :7 6:1y 25:8 5:4 29:4 3:1 21:9 3:4 14:0 0:3y 14:2 7:4 c=0.1 18:3 6:0 26:4 5:4 28:7 2:8y 21:9 4:0 13:9 0:2y 12:5 7:2 c= :7 6:3 27:0 5:0 31:2 2:6 23:0 3:9 14:5 0:2? y 14:0 7:2 Figue 2: The aveage pecetage eo ate of each of the citeias. We maked by? sigicat dieeces fom the stadad C4.5 (c4.5 -c 25) ad by y sigicat dieces fom the upued tee. (The sigicace level is 0.9 ad it uses a dieece of popotios test [Die96]) the paamete -m 1 ad -c 1. The upued tee had size 4567 (ad zeo eo o the taiig set). We a C4.5 with the default codece paamete (-c 25) which geeated a pued tee of size 3685 (ad oly 244 mislabels o the taiig set). Eve whe we chose a vey low codece level (-c 1) it geeated a pued tee of size 551 (with 3201 eos o the taiig set). I cotast, whe we used ou citeia, with c=0.7, as the theoy ecommeds, we ecoveed the best tee (of size 3!). (This best pued tee is also achieved usig c=0.5.) We cotiued ad checked the hypotheses geeated usig examples as a test set, i ode to test the geealizatio eo. Fo ou citeia we kow the exact eo, foty ve pecet. We tested the hypotheses geeated by C4.5, ad they had eo 49.5% (fo -c 25) ad 47.1% (fo -c 1). As expected, the esults show that the supeuous odes oly icease the geealizatio eo. Usig test of popotios [Die96], a 95% codece iteval aoud the best eo (45%) gives the iteval [44:3; 45:7], which implies that the deviatios, as expected, ae vey sigicat. We did ot limit ouself to aticial data, but cosideed also a vaiety of eal databases. I ode to pefom the compaiso we chose six databases, fou take fom the poject StatLog [Sta] ad two databases fom UCI Repositoy [MA94]. Sice we ae iteested i fuctios with boolea output, the databases all have to pedict a boolea output. The databases ae: 1. austalia { Austalia Cedit database. The aim is to decide whethe to appove a cedit cad applicatio. Thee ae 690 examples. Each example has foutee attibutes, six cotiuous ad eight categoical. 2. diabetes { Pima Idias Diabetes database. The aim is to pedict whethe a patiet shows sigs of diabetes. Thee ae 768 examples. Each example has eight cotiuous attibutes. 3. gema { Gema Cedit database. Thee ae 1000 examples. Each example has twety attibutes, seve umeical ad thitee categoical. 4. heat { Heat Disease database. Thee ae 270 examples. Each example has thitee cotiuous attibutes. 5. adult { US Cesus Bueau database. The aim is to pedict whethe the salay of a peso is geate o less tha 50,000$. Thee ae examples. Each example cotais foutee attibutes (six cotiuous ad eight omial). 6. Ioosphee { Rada data about the fee electos i the ioosphee. The aim is to pedict if the stuctue is \good" o \bad". Thee ae 351 examples. Each example has thity fou cotiuous attibutes. I the expeimet we pefomed a 5-fold coss validatio. (Namely we split the iput to 5 equal pats, taied o fou pats ad tested o the fth. We did this ve times, oe fo each combiatio.) I Figue 1 we have the mea tee size that the vaious citeia have geeated. We cosideed the upued tee, the dieet codece paametes to C4.5 (1, 10 ad 25) ad fou settigs of ou citeia (deoted by c= #, whee # is the costat we used fo c). It is vey clea that i ou citeia, as we icease the paamete c we ae gettig smalle ad smalle tees,

6 ad by vey sigicat factos. Iitially, whe c=0.05, thee is vey little puig. As we get the paamete to c=0.5 the tees that ae geeated ae extemely small. I Figue 3 we plot the aveage tee size as a fuctio of c fo the database adult. The eo ate of the dieet citeia appeas i Figue 2. I ode to test the sigicace of the esults we used the test of popotios [Die96] with codece 90%. We compaed the esults to the upued tee ad the C4.5 with the default settig (i.e., c4.5 -c 25. The esults show that the settig of ou paamete to c=0.5 is too pessimistic, ad it ove-pues. The othe ifeio citeia is the upued tee, hee clealy we see that puig helps. All the othes seem competitive with each othe. I Figue 4 we plotted the eo as a fuctio of the paamete c used i ou citeia. As expected, the behavio is as follows. Iitially, the upued tee, has a lage eo, the, as moe puig is pefomed, the eo dops to some miimal value (aoud c=0.1, i ou case). Fom this miimum value, as we pefom moe puig, the eo stats to icease as we pue moe ad moe. Ackowledgemets I would like to thak Willia W. Cohe ad Thomas G. Dietteich fo poitig me to the elevat liteatue. This eseach was suppoted i pat by a gat fom the Isael Sciece Foudatio. Refeeces [AHM95] Pete Aue, Robet C. Holte, ad Wolfgag Maass. Theoy ad applicatios of agostic PAC-leaig with small decisio tees. I The 12th Iteetioal Cofeece o Machie Leaig, pages 21{30. Moga Kaufma, [BB94] Maco Bohaec ad Iva Batko. Tadig accuacy fo simplicity i decisio tees. Machie Leaig, 15:223{250, [BFOS84] Leo Beima, Jeome H. Fiedma, Richad A. Olshe, ad Chales J. Stoe. Classicatio ad Regessio Tees. Wadswoth Iteatioal Goup, [Die96] Thomas G. Dietteich. Statistical tests fo compaig supevised classicatio leaig algoithms. mauscipt, [Koh96] Ro Kohavi. Scalig up the accuacy of aive-bayes classies: a decisio-tee hybid. I Poceedigs of the Secod Iteatioal Cofeece o Kowledge Discovey ad Data Miig, page to appea, [MA94] P. M. Muphy ad D. W. Aha. Uci epositoy of machie leaig databases. mlea /MLRepositoy.html, [Mi87] Joh Miges. Expet systems { ule iductio with statistical data. Joual of the Opeatioal Reseach Society, 38:39 { 47, [Mi89] [Nib87] [Qui86] Joh Miges. A empiical compaiso of puig methods fo decisio tee iductio. Machie Leaig, 4:227 { 243, T. Niblett. Costuctig decisio tees i oisy domais. I I. Batko ad N. Lavac, editos, Pogess i Machie Leaig{ Poceedigs of EWSL 87: 2d Euopea Wokig Sessio o Leaig, pages 67{78, Bled, Yugoslavia, May J. R. Quila. Iductio of decisio tees. Machie Leaig, 1:81{106, [Qui87] J. R. Quila. Simplifyig decisio tees. Iteatioal Joual of Ma- Machie Studies, [Qui93] J. R. Quila. C4.5: Pogams fo machie leaig. Moga Kaufma, [Sta] StatLog. Compaative testig ad evaluatio of statistical ad logical leaig algoithms fo lage-scale applicatios i classicatio, pedictio ad cotol. ftp:ftp.cc.up.pt/pub/statlog /datasets, (See also: Machie Leaig, Neual ad Statistical Classicatio, ed. Michie, Spiegelhalte ad Taylo). [VC71] V. N. Vapik ad A. Ya. Chevoekis. O the uifom covegece of elative fequecies of evets to thei pobabilities. Theoy of Pobability ad its applicatios, XVI(2):264{280, 1971.

7 tee size c paamete Figue 3: The aveage tee size as a fuctio of the paamete c fo the database adult Figue 4: The aveage eo as a fuctio of the paamete c fo the database adult.