Anomalies as Precursors of Field Failures

Transcription

1 nomaies as Precursors of Fie Faiures ebastian Ebaum, atya Kanuri Department of omputer cience an Engineering University of Nebraska - Lincon Lincon, Nebraska {ebaum,skanuri}@cse.un.eu nneiese mscher nrews choo of Eectrica Engineering an omputer cience Washington tate University Puman, W aanrews@eecs.wsu.eu bstract Reproucing an earning from faiures in epoye software is costy an ifficut. Those activities can be faciitate, however, if the circumstances eaing to a faiure are propery capture. In this work, we empiricay investigate how various anomay etection schemes can serve to ientify the conitions that precee faiures in epoye software. Our resuts expose the traeoffs between ifferent etection agorithms appie to severa types of events uner varying eves of in-house testing. 1. Introuction Faiures occurring in the fie can be costy an are often har to reprouce. To enabe an acceerate the faiure reprouction an faciitate ebugging activities, some information is usuay gathere at the time of the faiure. Netscape, Microsoft, an nome, for exampe, have impemente ifferent types of automate faiure reports that take a snapshot of the system at the time of the faiure to assist in assessment an ebugging. This approach, athough hepfu, cou be greaty enhance if the circumstances eaing to a faiure are capture, proviing a richer set of reevant cues about the faiure [3]. One of the ifficuties in capturing such information is etermining when to begin the coection process. On one extreme, the common practice is to trigger coection when a fata faiure occurs. This minimizes the coection overhea at the cost of vauabe information. On the other extreme, one cou capture a events since the beginning of execution, which wou provie more information but cause an important execution overhea [12]. nticipating the occurrence of a faiure cou provie a means to etermine more efficienty when to trigger the ata coection. Furthermore, anticipating the occurrence of ifferent types of faiures cou make this process more effective by taioring what ata is capture base on the particuar faiure scenario. To anticipate faiures we propose to continuay examine the system fie behavior in search of instances that eviate from a baseine behavior experience in-house. imiar approaches have been consiere in tangentia efforts. For exampe, anomay etection is being empoye successfuy on fie ata to etect potentia security threats or intrusions [18], to provie continuous assessment of fie ata quaity [19], an to track program invariants for automatic ebugging [3, 11]. Our conjecture is that there are anomay etection mechanisms (anaogous to the ones just mentione) to effectivey anticipate fie faiures. If we inten to utimatey use anomaies as preictors of fie faiures, we must first improve our unerstaning of their reationship. It is possibe that some of the faiures are not precee by anomaies. It is aso possibe for some anomaies to occur in the absence of a faiure. Finay, it is aso possibe that we fin meaningfu reationships between certain types of anomaies an certain faiures. The goa of this work is then to achieve a better unerstaning about the reationship between anomaies an faiures by evauating the preictive capabiities of various anomay types an agorithms in the context of faiure anaysis. To procee with our assessment we wi efine ifferent categories of observabe events an anomay etection agorithms an etermine empiricay if an which of those anomaies are effective preictors of faiures. Next section efines the scope of the probem, the event types, an introuces the basic agorithms. ection 3 presents the experimenta subject, esign an threats to vaiity. ection 4 presents the anaysis an interpretation. We summarize our finings in ection 5.

2 2. Probem efinition n anomay can be efine as any conition that eviates from expectations erive from requirements specifications, esign ocuments, stanars, or from someone s perceptions or experiences [1]. We wi empoy this efinition to introuce the various components an issues of an anomay etection scheme an its reationship to faiures Observabe Dimensions The first impication of this efinition is that there exist a set of observabe events refecting the system conition at a given point in time. uch observabe events range from the execution of an entity (e.g., function, statement, path, user operation) to configuration changes (e.g., changes in oae moues, moification in environment variabes). evera factors affect the ecision of what type of events are to be monitore. First, the seection of events must be associate with the utimate outcome (faiure preiction in our case). The secon factor is the cost of coecting the events. Event coection usuay impies aitiona system preparation (e.g., coe instrumentation) an execution overhea. uch costs can infuence not ony what to monitor but aso how often to monitor. Thir, the size of the event group to be monitore can efine what is to be seecte. arger number of events to be monitore, (aso known as the aphabet size) can aversey affect the anomay etection agorithm sensitivity because it makes it harer to provie an appropriate training set. Fourth, the reguarity of the events. n anomay etection approach is ess ikey to be effective on a system with a high egree of variabiity where the norma behavior is harer to efine. onsiering these factors, we investigate four types of events: function execution, user operations invocations, configuration changes, an moification in environment variabes. These constitute observabe imensions that have been known to reate to faiures, that can be coecte efficienty, an that present varying size an reguarity which wi et us anayze their traeoffs Detection gorithms The anomay efinition aso states that an anomay is present when eviations are etecte aong an observabe imension. nomay etection agorithms aim at etecting surprising events or sequences of events. In our work, we have concentrate on three genera types of agorithms. We introuce these agorithms in increasing orer of sensitivity. Binary-base Detection gorithm. iven a set of norma events E, this agorithm etects an anomay when an event outsie E is execute in the fie. For exampe, if a system has five functions,b,,d, an E, an ony,, an E are teste, the agorithm wou point out an anomay when B or D are execute in the fie. This type of agorithm is aso referre to as foreign-symbo anomay etection agorithm, which raises a fag when a symbo not incue in a training set is execute. Frequency-base Detection gorithm. iven a set of a observabe events E an their associate maximum an minimum expecte frequencies of execution, this agorithm etects an anomay if an event execution frequency after reease is outsie the efine ranges. ince an event execution frequency grows with onger executions, we constrain the efinition an measurement of frequencies to fixe-size observabe winows. s a resut, the ranges are compute base on the behavior observe in each execution winow. For exampe, et the set Freq represent the set of ranges observe uring pre-reease testing for each event e E. For a winow size fixe at n, this agorithm ivies each fie execution into winows of n events each. In each of these winows, the frequency of occurrence of each e E is compute an then compare against the expecte ranges efine by Freq. The frequencies that are outsie the expecte range are fagge as anomaies. equence-base Detection gorithm. iven a set of norma events E, an a set T of norma sequences (of events in E), this agorithm points out an anomay when a new interevent transition is observe. This agorithm uses a simpe state machine to maintain transitions efine as norma, an it etects anomaies when new eges are traverse or new noes are create by a fie execution. For exampe, if 80% of the user operations have been execute uring in-house testing, the set E hos the teste operations an set T contains the transitions observe between any two operations in E. Then, for a pair of sequentiay execute operations Op i an Op i+1, the agorithm points out an anomay if, either Op i / EorOp i+1 / E, or the transition Op i Op i+1 / T. This agorithm fits into a more genera type of agorithm cae foreign n-gram anomay agorithm [17] which incues sequences of size n. iven the arge size of events we gather, we have constraine our agorithm to sequences of size Estabishing a Baseine Having estabishe the type of observabe events an the anomay etection agorithms (that constitute an anomay etection scheme), we must now efine what constitutes norma or nomina behavior. ince our work concentrates on faiures an anomaies occurring in the fie, our expectations for norma behavior wi be constructe base on pre-reease behavior. More specificay, our stuy uses the behavior expose by integration testing to efine what constitutes baseine behavior (etais about the impementation an impact of this ecision are presente in ection 3).

3 Using this baseine, fie ata can be monitore to ientify incients outsie the norm. Note that the eviations reporte from the fie may not necessariy resut from a faiure scenario. nomaies can aso be etecte in nonfaiure scenarios, either because the system s norma behavior was not characterize fuy uring in-house testing (e.g., the testing activity i not represent the behavior from the fie) or because a rare execution scenario happene that is not necessariy fauty. From a faiure anaysis perspective, a goo anomay etection scheme wou be sensitive to faiure-scenarios, but it wou not etect abnorma eviations in non-faiure scenarios (ow fase-positives) Research Questions The foowing research questions refine the previous notion an wi serve to guie our stuy: RQ1: How consistent is the presence of each type of an anomay uner faiure an non-faiure scenarios? RQ2: What combinations of anomay etection agorithms an observabe imensions are more effective at preicting faiures? RQ3: What is the effect of test suite potency in the accuracy of the agorithms? RQ4: What traeoffs are to be consiere to increase the sensitivity of the anomay etection scheme? 2.5. Reate Work nomay etection constitutes a core technoogy appie to severa omains. For exampe, anomay etection has been wiey use in the intrusion etection fie for severa years [4, 9, 16, 18]. Recenty, we have aso observe severa efforts attempting to migrate some of those anomay etection schemes to assist other fies such as fie ata quaity assessment [19] an faut isoation [11]. Our work beongs to these atest groups, trying to appy the existing anomay etection schemes to assist faiure anaysis activities an more specificay, as precursors of fie faiures. We aso exten previous research efforts in faiure pattern anaysis where the goa is to ientify whether a fie faiure has areay been observe (an its associate faut fixe) [15]. Our approach, however, aims to etect anomaies at execution time (before a potentia faiure occurs) instea of trying to match the symptoms a posteriori. The importance of anomay etection technoogy has e to the eveopment of numerous approaches incuing machine earning techniques [14], statistica moes [17], an pattern base ientification mechanisms [20]. The work presente in this paper ony begins to expore the most simpe approaches. Our work is aso reate to previous efforts to estimate faut content uring an after epoyment. These efforts can be cassifie into two groups accoring to their approaches to bui preiction moes. The first group empoys historica ata to ientify, for exampe, faut-prone components either within the same reease or between reeases [5, 10, 13, 23]. The secon approach empoys ata avaiabe ony from the current reease [6, 7, 21, 22] to estimate, for exampe, the efects remaining in a system after verification. Our approach iffers in that the estimation targets faiures, which impies that the assessment is performe ynamicay on the system behavior. 3. Experiment etup We begin this section by escribing the subject program, the epenent an inepenent variabes an the metrics use to quantify them. Later we escribe the esign of the experiment an iscuss the threats to experimenta vaiity ubject For the experiment to be meaningfu, the subject program must be: (1) as representative as possibe of rea wor programs, (2) generate events on which we can etect anomaies, an (3) have freey avaiabe source coe to aow experimentation. Pine, a Program for Internet News an Emai, met those requirements. The experiment empoye 5 versions of Pine whose characteristics appear in Tabe 1. In aition to the program, we require a test suite that exercise Pine s functionaity. That suite is important because it constitutes the basis for eriving faiure an nonfaiure scenarios (expaine in the next section). To construct test cases representative of those that might be create in practice for these programs, we aapte a process use in our previous stuies. We use the avaiabe ocumentation on the programs to erive informa specifications. We then empoye the category partition metho to construct a suite of test cases that exercises at east one of each parameter, specia effect, an erroneous conition affecting program behavior. This process resute in a test suite of 289 test cases, that exercise approximatey 60% of Pine s functions (3r coumn of Tabe 1) Faiure an Non-Faiure cenarios The experiment require execution scenarios that refect the software behavior in the fie, as we as those that refect in-house testing. iven our premise that in-house testing is possiby incompete (i. e. oes not capture the observe behavior in the fie fuy), the in-house test scenarios are a subset of the fie scenarios. In aition, ony

4 Tabe 1. haracteristics of Pine Version Number of Functions Functions overe(%) Functions hange Lines of oe Number of Fauts (from previous version) v (61%) v (62%) v (59%) v (57%) v (56%) fie scenarios can exhibit faiures, if we assume that faiures uring in-house testing were repaire. This eas to the foowing reationship between fie scenarios an in-house scenarios. Let the set of fie scenarios be T = T f T n ; where T f are the fie scenarios eaing to faiure an T n are the scenarios executing without faiure. Then, the set of in-house scenarios (tests) is T T n. This refects the case when fie scenarios expose behavior that was not fuy teste inhouse. This efinition aso assumes that a in-house scenarios eaing to faiure were fixe before reease an no changes were mae to the epoye software, so running them again either in-house or in the fie oes not ea to faiure To test whether anomaies an faiures are reate requires efinition of a set T f, fie-execution scenarios that resut in faiures. For this purpose, Pine was seee with artificia fauts (coumn 5 in Tabe 1) 1. faut-seee version of Pine was consiere to fai when it prouce a ifferent output than the version not seee with fauts. Test cases that prouce faiures are faiure scenarios (T f ), those that o not are non-faiure scenarios (T n ) Variabes an Metrics Depenent Variabes. We use two metrics to capture the reationship between anomaies an faiures. The first metric captures the percentage of fie scenarios in which an anomay occurs at east once. We ca this metric sensitivity an use it to compare the performance of an anomay etection scheme uner faiure an non-faiure scenarios. ensitivity can be efine formay as foows. Let be a set of fie scenarios with a carinaity of Tot : = {s 1,..., s Tot }, where s i is the i th scenario of. Let E si be the set of events in the scenario s i. Then, a scenario is anomaous if nom si is 1, where nom si is efine as, 1, if an event e E si such that nom si = e constitutes an anomay 0, otherwise 1 Detais about the empoye seeing process can be foun in [8]. (1) The number of scenarios that have at east one anomay, nom, can be efine as nom = Tot i=1 Finay, sensitivity can be efine as nom si (2) ensitivity = nom (3) Tot goo anomay etection scheme wou have high sensitivity to faiure scenarios an ow sensitivity to non-faiure scenarios. secon metric quantifies the sensitivity ivergence between faiure an non-faiure fie scenarios. We ca this metric accuracy, an efine it as the ifference between the average sensitivity for fie faiure scenarios an the average sensitivity for fie non-faiure scenarios. ccuracy vaues fa in the interva [-1,1]. Formay, given F (the sensitivity compute over a set of faiure scenarios) an NF (the sensitivity compute over a set of non-faiure scenarios), accuracy can be efine as ccuracy = F NF (4) Inepenent Variabes. The inepenent variabes of this experiment are the observabe imension, the anomay etection agorithm, an the test suite potency. n observabe imension is the set of events, that can be trace, refecting the system at a given point in time(ection 2). n anomay etection agorithm aims at etecting eviations aong such observabe imensions (ection 2). The imensions consiering configuration changes an moification in environment variabes (ection 2) i not show any anomaous behavior in faiure scenarios, an hence, were iscare. This eaves function execution an user operation invocations from the origina observabe imensions; we ae a hybri imension that jointy consiers execution an user operations. Test uite Potency represents the estimate proportion of the system s fie execution behavior that is expose by the test suite uring in-house integration testing. We have efine potency as a percentage of the test cases in T n that

5 comprise the test suite. We expore 5 test suites with potencies 0.5, , 0.99, The test cases for each suite were ranomy chosen from T n. test suite potency of 1 impies that our test suite refects 100% of the fie scenarios that i not cause a faiure Design an Impementation Each faut-free version of Pine was first execute to construct its corresponing orace. Then, each faut-seee version of Pine was instrumente to capture the events across the target imensions an the test suite was execute. We then cassifie the execute test cases as faiure an nonfaiure scenarios epening on whether the seee faut cause a faiure or not. Last, we use the coecte event ata to run our ifferent anomay etection agorithms, compute the metrics whie simuating ifferent test suite potencies, an etermine their reationship with each type of scenario. Note that, athough our anomay etection agorithms were execute off-ine to minimize the costs of event coection (over 90 hours), they cou be incorporate into the target program without oss in generaity. The experiment format consists of a fu factoria esign with 3 factors (imension, anomay etection agorithm an test suite potency). There are 3 eves of imension (function execution, user operation, joine), 3 ifferent anomay etection agorithms (binary, frequency, sequence), an 5 eves of test suite potency (0.5, 0.75, 0.95, 0.99, an 1.0). In each ce we have a tota of 5 observations, one per version which resuts in a tota of 225 ata points. The anaysis of these ata points with respect to the two epenent variabes is presente in ection Threats to Vaiity Threats to interna vaiity are infuences that can affect the epenent variabes with respect to causaity, without the researcher s knowege. In our stuy, the simuation of faiures through seee fauts may have biase the resuts if the fauts inserte were such that certain agorithms or imensions were benefite. This threat was partiay controe by having a grauate stuent, who was unaware of the experiment, see fauts in Pine foowing a set of estabishe guieines. In aition, the investigator who conucte the experiment ha no knowege of the kins of fauts that were seee. nother threat to interna vaiity is the assumption that we can exacty simuate a subset of the post-reease behavior of software uring pre-reease testing. We controe this threat by generating the execution scenarios that exercise iniviua functionaity so that they cou be easiy reprouce an sti ho our assumption about the meaning of an in-house test. Finay, the infrastructure require to execute this stuy invoves many toos. though some of the toos were use an teste in previous experiments, some new toos were written to gather, fiter an process trace information. We manuay verifie resuts an ran a series of test cases to obtain confience in the new toos correctness. Threats to construct vaiity are significant when the measurement instruments o not capture the concepts they are suppose to capture. Quantifying anomay etection systems is commony chaenging because their effectiveness epens on how we norma conitions are ientifie. For exampe, intrusion etection systems base on anomay etection strugge constanty to etermine what constitutes norma user behavior. In our omain, the norma conitions are efine by the in-house testing activity, where anything that was not teste constitutes an abnorma behavior. Our manipuation of test suite potency et us to further expore the variabiity cause by the test suite characteristics. ti, system behavior variabiity can affect the effectiveness of the anomay etection scheme. In aition, there are other measures to quantify the reationship between anomaies an faiures (e.g, time for reproucing a faiure) that were not irecty appicabe to our stuy, but shou be investigate in a next phase. Threats to externa vaiity are conitions that imit researcher s abiity to generaize the resuts of an experiment to inustria practice. iven that no singe program can be representative of a possibe programs, we certainy are imite in our abiity to generaize from our resuts to a possibe programs. However, resuts wou ikey be appicabe to simiar programs. The same appies to the specific efinition of faiure an non-faiure scenarios. 4. Resuts an naysis This section anayzes the experimenta resuts from severa perspectives. Initiay, we use escriptive statistics to expore the sensitivity of faiure an non-faiure scenarios. Then, we investigate the agreement between our anomay etection schemes an the occurrence of faiures. Last, we anayze the impact of agorithm, imension an test suite potency on accuracy Exporatory naysis of ensitivity There are four possibe outcomes of our anaysis on faiure scenarios an non-faiure scenarios an anomaies: (1) an anomay precees a faiure, (2) an anomay is etecte without a subsequent faiure (a fase positive), (3) a faiure occurs without a preceing anomay (fase negative), an (4) no anomay was etecte for a non-faiure scenario. To start examining these possibiities we generate three pots epicting sensitivity vaues. Each pot anayzes one variabe at a time, epicting F on the x-axis an NF on

6 Binary base Frequency base equence base nomay Detection gorithms F NF Figure 1. ensitivity of F an NF by nomay Detection gorithm Function a Function a User Input User Input Dimension F NF Figure 2. ensitivity of F an NF by Dimension

7 NF Test uite Potency F Figure 3. ensitivity of F an NF by Potency Leve the y-axis. Each observation in the figures resuts from the sensitivity vaues for faiure an non-faiure scenarios uner each combination of test suite potency, etection agorithm, observabe imension, an version of the program. The first scatter pot in Figure 1 focuses on the anomay etection agorithms. We conjecture that, per efinition, a sequence-base etection agorithm wou be more sensitive than a frequency-base etection agorithm, which in turn wou be more sensitive than the binary-base etection agorithm. though we are aware of the noise in the ata (it umps together a test potencies an imensions), we wante to investigate whether that conjecture he. Upon observing these pots, we note that the sequence-base anomay etection agorithm is inee the most sensitive, with its observations groupe at the bottomright corner, with F ranging from 0.67 to To a esser extent, the sequence base agorithm was aso the most sensitive to anomaies with NF surpassing 0.5 in 2 instances. The sensitivity of the other two agorithms aso foows the conjecture, with binary-base being the east sensitive to anomaies uner both types of scenarios. The secon pot in Figure 2 iscriminates the ata excusivey base on imension. In this graph, we cou not extract any cear pattern cause by shifts in imension. The thir scatter pot, Figure 3, iustrates the sensitivity vaues for five test suites with ifferent potencies. We conjecture that, as test suite potency increases an more of the fie behavior was expose in-house, NF wou ecrease. In other wors, the more we know about the behavior of the program before reease, the better we are at etecting anomaies that are truy associate with faiures. The scatter pot shows that, as the test suite potency increases, the number of fase positives ecreases. For instance, for a test suite potency of 0.5, NF ranges from 0.10 to 0.54 whereas for a test suite potency of 1 (iea case), NF is aways. We cou not fin, however, a cear pattern for F perhaps because this graph umps together other factors such as etection agorithm an imension. Overa, we observe that an anomay etection scheme which empoys a highy sensitive agorithm, wi aso have to effectivey ea with the number of fase positives it generates. One viabe way to reuce the number of fase positives is to better characterize the system behavior by empoying a test suite of higher potency. For instance, in Figure 3, a test suite potency of 0.99 has a NF of 0.16 whereas a test suite potency of 0.5 has a worse NF of Hence, in orer to obtain higher sensitivity in faiure scenarios an ower sensitivity in non-faiure scenarios, it is essentia to empoy a combination of a sensitive agorithm an a highy potent test suite.

8 Kappa Kappa Kappa Tabe 2. Faiure Preiction from nomaies nomay Base Preiction Non Faiure Faiure um Prior Faiure p 11 p 12 p 1+ Knowege Non-Faiure p 21 p 22 p 2+ um p +1 p +2 Binary base bin 4.2. greement naysis The patterns observe in the previous section seem to inicate that certain agorithm etection schemes can be effective at preicting faiures. The goa of this section is to evauate to what egree a preiction system that, uses the occurrence of anomaies to preict ater faiures, is in agreement with an assessment that correcty cassifies faiures. We wi empoy the kappa statistic [2] to etermine whether the preictor (here: anomaies) eve of agreement with the outcome it preicts (here: faiures) excees chance. Tabe 2 shows that test scenarios can be categorize by how each moe cassifies them: p 11 is the proportion of test scenarios categorize as an anomay as we as a faiure, p 22 is the proportion of test scenarios showing neither anomay nor faiure, p 12 is the proportion of test scenarios showing faiure without a preceing anomay, an p 21 is the proportion of test scenarios showing an anomay that oes not ea to a faiure. The ata efine in Tabe 2 is use to compute the agreement inex as foows. Let P be the proportion where there is an agreement. Then, P = 2 p ii (5) i=1 The proportion of agreement by chance, P E is efine as P E = 2 p i+ p +i (6) i=1 Now, the kappa agreement inex can be compute as Frequency base equence base freq seq κ = P P E 1 P E (7) Vaues of the kappa statistic are often interprete in the scae suggeste by tman [2], which is given in Tabe 3. Figure 4 shows the kappa vaues as a function of test suite potency. Each kappa vaue in the three graphs is compute for a combination of imension an potency, for a specific anomay etection agorithm, an average across 5 versions. The observations of ifferent imensions in each pot are represente using ifferent styes to assist in the interpretation of its impact. The graphs suggest that the imension Function a User Input eas to the highest kappa vaues, regaress Dimension Function a Function a User Input Test uite Potency User Input Figure 4. greement Inices for nomaies an Faiures

9 ccuracy Binary base bin Tabe 3. The tman Kappa cae Kappa tatistic trength of greement [-] Poor (-] Fair (-] Moerate (-] oo (-1.00] Very oo Frequency base freq of the anomay etection agorithm. The sequence-base anomay etection agorithm has the highest kappa vaues of a three agorithms. For test suite potencies of at east 0.75, this agorithm wi give at east a Moerate strength of agreement, whie for greater potencies the agreement was oo or Very oo. Empoying ifferent imensions seeme to have ha a sma impact uner the sequence-base agorithm; the same is not true uner the other two agorithms where the user-input imension consistenty ranke one eve beow the best. Overa, we observe an increasing eve of agreement as the test suite potency increase, irrespective of the agorithms an imensions use, athough the rate of increase varie. ccuracy ccuracy equence base Dimension seq Test uite Potency Function a Function a User Input User Input Figure 5. ccuracy Box Pots 4.3. naysis of ccuracy The agreement resuts from the previous section are encouraging. However, we sti o not know to what extent the variabes we contro (agorithm, imension, test suite potency) impact the effectiveness of the anomay etection schemes. This section anayzes the reationship between our inepenent variabes an accuracy empoying box pots an then an anaysis of variance. Figure 5 presents 3 box pots, one per agorithm. Each one iscriminates for imension (renere in ifferent styes) an test suite potency (on the x-axis), whie proviing a concise representation for the istribution of accuracies across ifferent versions. The centra ine in each box marks the mean vaue. The eges of the box mark stanar eviation, an the whiskers mark the range. These pots suggest that the mean accuracy of an anomay etection scheme empoying the Function a User Input imension is higher than for the other imensions; this tren can be observe across a agorithms. The variance of the accuracy vaues for Function a imension seems to be the highest, which cou be attribute to its arge number of istinct events an the probabiity that these events are anomaous (in both faiure an non-faiure scenarios) varies highy with the particuar execution scenario use. When consiering anomay etection agorithms, the box pots inicate that the sequence-base agorithm is the most accurate with a high of 0.93; the frequency-base agorithm top accuracy is 0.74 an the binary-base agorithm

10 maximum is It aso appears that the sequence base agorithm has a ower variance than the others as per the size of the boxes an the extension of the whiskers. Last, increases in test suite potency resute in greater accuracy, irrespective of the agorithm an imension epoye. We performe an anaysis of variance (NOV) to formay assess the previous observations on accuracy. Tabe 4 presents the resuts, incuing effects consiere, sum of squares, egrees of freeom, mean squares, F vaue, an p- vaue for each source of variation we manipuate. ince we set apha to 0.05, an the p-vaue represents the smaest eve of significance that wou ea to the rejection of the nu hypothesis, we reject the hypothesis when p is ess than apha. The resuts inicate that a three primary effects (imension, agorithm, test suite potency) significanty affect the accuracy of the anomay etection scheme to preict a faiure. The anaysis aso puts in evience that there are two significant interactions. The first one inicates that the accuracy of a seecte imension varies significanty epening on the chosen agorithm (e.g., in the pots we observe that the function-ca imension worke best uner the sequence-base agorithm but not as goo uner the binarybase one). The secon interaction points that the etection agorithm accuracy is significanty affecte by the eve of test suite potency (e.g., in the pots we observe that the rate of improvement as test suite potency increase change epening on the agorithm). 5. oncusions an Future Work We have presente an empirica stuy exposing the traeoffs associate with ifferent anomay etection agorithms an imensions, an the importance of test suite potency. More specificay, we stuie whether anomaies are precursors to faiures, an how this reationship is affecte by the choice of anomay etection agorithm, observabe imension an test suite potency. s we have pointe out in the threats to vaiity, this empirica stuy has severa imitations. Keeping these imitations in min, we raw certain observations from this work. First, our resuts inicate that, as the test suite potency improves, the number of fase positives ecreases. This impies that, the power of the anomay etection scheme is enhance as the test suite better characterize the system spectrum of behavior. This is true regaress of the anomay etection scheme constituents. econ, our resuts showe that the sequence-base anomay etection agorithm is more sensitive an accurate than the other agorithms. In fact, its highest accuracy shows that this reativey simpe agorithm can be very effective (as per kappa) at preicting an impening faiure. Thir, we earne that certain imensions were prove futie (an were iscare) for our particuar subject emphasizing the fact that, the choice of the imension shou be mae after carefu consieration. Engineers attempting to insert anomay etection schemes into their software to assist in the reprouction an anaysis of fie faiures must be aware of these impications to minimize the risk of epoying an expensive infrastructure with ow return-vaue or of setting unreaistic expectations ue to poor system behavior characterization. From our resuts, we see severa avenues for future work. First, to aress the questions of whether our resuts generaize, further stuies are necessary. We pan to exten this stuy on aitiona systems. econ, we wi evauate aitiona anomay etection schemes that empoy more compex an sensitive agorithms an compementary imensions to better represent program behavior. The anomay etection iterature is quite rich in agorithms that we have not begun to expore. Last, we nee to start transitioning from experiments to case stuies performe with software engineers to assess whether the reationships we are starting to unerstan transate into concrete gains for the practitioners eaing with fie faiures. cknowegements This work was supporte partiay by the NF Information Technoogy Research program uner wars R to University of Nebraska, Lincon. We thank ameera Rey for heping prepare the pine subject for experimentation. References [1] oftware-engineering-tanars-ommittee of the IEEE omputer ociety, IEEE tanar assification for nomaies. The Institute of Eectrica an Eectronic Engineers, Inc., 345 East 47th treet, New York, NY , U, [2] D. tman. Practica tatistics for Meica Research. hapman-ha, [3]. Z. B. Libit,. iken. Distribute program samping. In Proceeings of the M IPLN 2003 onference on Programming Language Design an Impementation, pages 9 11, an Diego, aifornia, jun [4] D. Barbara, N. Wu, an. Jajoia. Detecting nove network intrusions using bayes estimators. In Proceeings of the First IM Internationa onference on Data Mining, hicago, U, pri [5] V. Basii an B. Perricone. oftware errors an compexity: n empirica investigation. ommunications of the M, 27(1):42 52, January [6] L. Brian, K. E. Emam, B. Freimut, an O. Laitenberger. comprehensive evauation of capture-recapture moes for estimating software efect content. IEEE Transactions on oftware Engineering, 26(6): , June 2000.

11 Tabe 4. NOV on ccuracy - ignificant effects are emphasize in itaics. Effect.. D.F. M.. F p Dimension nomay Detection gorithm Test uite Potency Dimension*nomay Detection gorithm Dimension*Test uite Potency nomay Detection gorithm*test uite Potency Dimension*nomay Detection gorithm*test uite Potency Error [7]. Eick,. Loaer, M. Long, L. Votta, an. Wie. Estimating software faut content before coing. In Proceeings of the 14th Internationa onference on oftware Engineering, pages 59 65, [8]. Ebaum,.. Maishevsky, an. Rotherme. Test case prioritization: famiy of empirica stuies. IEEE Transactions on oftware Engineering, 28(2): , Feb [9]. Forrest,. Hofmeyr,. omayaji, an T. Longstaff. sense of sef for unix processes. In In Proceeings of the 1996 IEEE ymposium on ecurity an Privacy, pages IEEE omputer ociety Press, [10] T. raves,. Karr, J. Marron, an H. iy. Preicting faut incience using software change history. IEEE Transactions on oftware Engineering, 26(7): , Juy [11]. Hanga an M. Lam. Tracking own software bugs using automatic anomay etection. In Proceeings of the Internationa onference on oftware Engineering. IEEE-M, May [12]. Kanuri an. Ebaum. n empirica stuy of tracing techniques from a faiure anaysis perspective. In Proceeings of the Internationa ymposium of oftware Reiabiity Engineering, pages , nov [13] T. Khoshgoftaar an J. Munson. Preicting software eveopment errors using compexity metrics. Journa on eecte reas in ommunications, 8(2): , Feb [14] T. Lane an. Broey. n appication of machine earning to anomay etection. In In Proceeings of the 20th Nationa onference on Nationa Information ystems ecurity, voume 1, pages , aithesburg, MD, Nationa Institute of tanars an Technoogy. [15] I. Lee, K. Iyer, an. Mehta. Ientifying software probems using symptoms. In Procs. of the Internationa ymposium on Faut- Toerant omputing, pages , pri [16] M. Mahoney an P. han. Learning nonstationary moes of norma network traffic for etecting nove attacks. In Proceeings of the Internationa onferemce Knowege Discovery an Data Mining, pages M, [17] R. Maxion an K. T. Benchmarking anomay-base etection systems. In Internationa onference on Depenabe ystems an Networks (DN 2000). IEEE, June [18] P. Neumann an P. Porras. Experience with EMERLD to ate. In Proceeings of the 1st UENIX Workshop on Intrusion Detection an Network Monitoring, pages Usenix, apri [19] O. Raz, P. Koopman, an M. haw. emantic anomay etection in onine ata sources. In Proceeings of the Internationa onference on oftware Engineering. IEEE-M, May [20] R. ekar, M. Benre, D. Dhurjati, an P. Boineni. fast automaton-base metho for etecting anomaous program behaviors. In Proceeings of the 2001 IEEE ymposium on ecurity an Privacy. Institute of Eectrica an Eectronics Engineers Inc., May [21]. V. Wie an L. Votta. ssessing software esigns using capture-recapture methos. IEEE Transactions on oftware Engineering, 19(11): , November [22]. Wohin an P. Runeson. n experimenta evauation of capture-recapture in software inspections. Journa of oftware Testing, Verification an Reiabiity, 5(4): , [23] T. Yu, V. hen, an H. Dunsmore. n anaysis of severa software efect moes. IEEE Transactions on oftware Engineering, 14(9): , eptember 1988.