B.H. Far

Transcription

1 SENG 521 Software Reliability & Software Quality Chapter 13: Applying Failure Data to Guide Decision Department t of Electrical l & Computer Engineering, i University it of Calgary B.H. Far (far@ucalgary.ca) far@ucalgary.ca 1

2 SRE: Process (Review) 5 steps in SRE process: Define necessary reliability Develop operational profiles Prepare for testt Execute test Apply failure data to guide decisions Define Necessary Reliability Develop Operational Profile Prepare for Test Execute Test Apply Failure Data to Guide Decisions far@ucalgary.ca 2

3 Types of Decisions Decisions i related dto certification test t Accept/reject an acquired component Accept/reject a subsystem, OS, hardware, etc. Decisions related to reliability growth test Guiding software development process Releasing the product Decision related to adequacy of tests far@ucalgary.ca 3

4 Three Techniques 1. Reliability Demonstration Chart based on inter failure times & target failure intensity (or MTTF) 2. Reliability growth analysis based on inter failure times/ failure count and failure intensity i (or MTTF) 3. Zero failure testing based on failure density far@ucalgary.ca 4

5 Section 1 Test Completion & Release Criteria: Certification far@ucalgary.ca 5

6 Review: FIO Failure intensity objective (F) Failure intensity objective is defined as failure per natural units, e.g. 3 alarms per 100 hours of operation. 5 failures per 1000 print jobs, etc. Mean time to failure (MTTF) Expected time that next failure will be observed. A 1 MTTF 1 therefore MTTF 1 t MTTF t 1 m m far@ucalgary.ca 6

7 Decision Making Risks Risk related to the measured entity measurement risk; main risk component: In multiple stakeholder decision making involving supplier and purchasers: Risk in assessing the entity which is actually wrong to be right ( purchaser s risk: ) Risk in assessing the entity which is actually right to be wrong ( supplier s risk: ) RDC can use these 3 risks far@ucalgary.ca 7

8 Using Measurement Risk Only Risk related to the measured entity measurement risk; main risk component: Failur re number Reject Accept ln 1 1 Decreasing brings the line closer to vertical: less risk makes the product more acceptable Normalized failure time Increasing brings the line closer to horizontal: more risk makes the product more rejectable far@ucalgary.ca 8

9 Reliability Demo Chart /1 An efficient way of checking whether the FIO (F) is met or not. It is based on collecting failure data at time points. Vertical axis: failure number (n) Horizontal axis: normalized failure data (Tn), i.e., failure time F or failure time / MTTF Figure from Musa s Book far@ucalgary.ca 9

10 Parameters Involved /1 Discrimination i i ratio (1<): Acceptable error in estimating failure intensity. Customer risk k(0< (0<<1 <1) : Probability bilit that t the developer is willing to accept of falsely saying the failure intensity objective is met (i.e., acceptance) when it is not. Developer risk (0<<1 <1) : Probability that the developer is willing to accept of falsely saying the failure intensity objective is not met (i.e., rejection) when it is. far@ucalgary.ca 10

11 Parameters Involved /2 For =10% and = 10% and =2 There is 10% risk () of wrongly gyaccepting the software when its failure intensity objective is actually equal or greater than twice ( =2) the failure intensity objective. There is 10% risk () of wrongly rejecting the software when its failure intensity objective is actually equal or less than half ( =2) the failure intensity objective. far@ucalgary.ca 11

12 Reliability Demo Chart /3 1 A ln B ln 1 A changes rapidly with customer risk () but very slightly with developer risk () B changes rapidly with developer risk () but very slightly with customer risk () B/ln γ A/(1-γ) far@ucalgary.ca 12

13 Reliability Demo Chart /3 Boundary between accept and continue regions T n A ln 1 1 n ( is the discrimination ratio) Boundary between reject and continue regions T n B ln n 1 1 ( is the discrimination ratio) far@ucalgary.ca 13

14 Reliability Demo Chart /4 Values of intercepts of boundaries with various horizontal and vertical axes 14

15 Reliability Demo Chart /5 Vl Values of fa and dbf for various consumer and supplier risk levels Table from Musa s Book far@ucalgary.ca 15

16 Reliability Demo Chart /6 When risk levels ( and ) decrease, the system will require more test before reaching the accept or reject regions, i.e., the continue region becomes wider. When discrimination ratio () decreases, the system will require more test before reaching the accept or reject regions, i.e., the continue region becomes wider and the slope of the boundary line tends towards vertical. far@ucalgary.ca 16

17 RDC: When and How When to use RDC? When failure data is limited to a few failures, time of failures are known, and one wants to find out what is the trend for reliability of the system. How to use RDC? Collect failure data (failure number and failure time). Identify the target MTTF and anticipated confidence levels. Draw the failure points on the graph and analyze the trend. far@ucalgary.ca 17

18 RDC: Example /1 Consumer risk = 5% Supplier risk = 5% Discrimination ratio = 2 Figure from Musa s Book far@ucalgary.ca 18

19 RDC: Example /2 Consumer risk = 1% Supplier risk = 1% Discrimination ratio = 2 Figure from Musa s Book far@ucalgary.ca 19

20 RDC: Example /3 Consumer risk = 0.1% Supplier risk = 0.1% Discrimination ratio = 2 Figure from Musa s Book far@ucalgary.ca 20

21 RDC: Example /4 Consumer risk = 10% Supplier risk = 10% Discrimination ratio = 1.2 Figure from Musa s Book far@ucalgary.ca 21

22 Example 1 Failure number Measure Normalized (million Measure transactions) (MTTF) F 4 failures/million transactions %10 %10 2 far@ucalgary.ca 22

23 Example 2 Failure number Measure (CPU hour) Normalized Measure (MTTF) F 0.1failures/CPU hour far@ucalgary.ca 23

24 Example 3 We are going to buy a new colour copier for our department. We have borrowed the copier for the test run and we are going to conduct certification test on it. Maker s data shows that we need to change the toner every 10,000 pages. We would like to have the system running with only one failure during the lifetime of a toner (i.e. one paper jam during the toner s lifetime). a) What shall be our failure intensity objective for the system? F = 1/10,000 pages far@ucalgary.ca 24

25 Example 3 (contd.) b)we observed that paper jams occurred at 4,000 pages, 6,000 pages, 10,000 pages, 11,000 pages, 12,000 pages and 15,000 pages of output. Does it meet our objective? far@ucalgary.ca 25

26 Example 3 (contd.) Because of failing the certification test we will reject the copier. far@ucalgary.ca 26

27 Example 3 (contd.) c) What if the paper jams occurred 11,000 pages, 12,000 pages and 15,000 pages of output? Does it meet our objective now? Not sure. We may need more testing. far@ucalgary.ca 27

28 Example 4 We have developed da program for a Web server with the failure intensity of 1 failure/100,000 transactions. The program runs for 50 hours, handling 10, transactions per hour on average with no failures occurring. How confident are we that the program has met its objective? Can we release the software now? far@ucalgary.ca 28

29 Example 4 (contd.) Answer The total number of transactions is 50 10,000 = 500,000 Therefore, normalized failure time is 5. Using the reliability demonstration chart, this point is well in the accept region, therefore the answer is YES and there is 10% risk of wrongly accepting or rejecting the software. far@ucalgary.ca 29

30 Example 4 (contd.) Suppose that ta new component tis added dto the program serially to make a new package. The failure intensity of the new component is 0.5 failures/100,000 transactions. The new package fails after 10 hrs. Can we release the new package now? Answer The new failure intensity for serial system λ = λ1 + λ2 = 1.5 failures per 100, transactions. Total number of transactions until failure is 10 10,000 = 100,000. Therefore, normalized failure is 1.5. Using the reliability demonstration ti chart, this point is in the continue region, therefore the answer is NO and we have to continue testing. far@ucalgary.ca 30

31 RDC: Benefits and Limitations RDC analysis is very time and cost efficient i way of analyzing the reliability of a system. It can be used as a demonstration of the reliability of the system to support decisions. A disadvantage of the RDC analysis is that it cannot come up with a quantitative number for the reliability (or availability) of the system under study. It can only demonstrate the trend of changes and how they affect the reliability of the system. Another disadvantage is that experimenting with different values of confidence levels l and MTTF (what-if scenarios) is possible but rather tedious. RDC open source project: far@ucalgary.ca 31

32 Section 2 Test Completion & Release Criteria: Reliability Growth far@ucalgary.ca 32

33 Types of Decisions Decisions related to certification test Accept/reject an acquired component Accept/reject a subsystem, OS or hardware Decisions related to reliability growth test Guiding software development process Releasing the product far@ucalgary.ca 33

34 Release Criteria Indicators from at tleast tthe following dimensions i should be considered together to get an adequate picture of the quality of the product. System stability, reliability, and availability Defect volume Our focus Outstanding critical problems Feedback from early customer programs Other quality attributes that are of specific importance to a particular product and its customer requirements and market acceptance (e.g., ease of use, performance, security, portability, etc.) From Dr. Kan s Book far@ucalgary.ca 34

35 Review: Test Completion Criteria Q. When is testing enough? Test phase time or resources are exhausted All black-box test cases are run White-box test coverage targets are met Rate of fault discovery goes below a target value Target percentage of all faults in the system are found Measured reliability of the system achieves its target value far@ucalgary.ca 35

36 Approach: Models Use tools such as CASRE to Plot the /F ratio vs. time to see the trend of changing. CASRE uses basic exponential and logarithmic Poisson and a few other models. The basic exponential model assumes finite failures in infinite time; the logarithmic Poisson model assumes infinite failures. In estimating the /F ratio, the basic exponential model tends to be optimistic (low); the logarithmic Poisson model tends to be pessimistic (high). far@ucalgary.ca 36

37 Release Criteria /1 The estimation of normalized failure intensity has a range of uncertainty. This is defined by confidence limit. For example, if the confidence limit is %90, one would expect the system meet its failure intensity objective with only %90 confidence. CASRE 2.0 does not compute confidence limits. far@ucalgary.ca 37

38 Release Criteria /2 How to include the confidence limit in estimation? An approximate approach can be devised: Consider sde terminating test for the esyse system when normalized failure intensity /F drops to 0.5 or less. Assume that %90 confidence interval for failure intensity near release is approximately equal to 0.5 to 1.5 times the target failure intensity F. 38

39 Release Criteria /3 Consider releasing the product when 1. Test (feature and load) terminated satisfactorily for the product itself 2. Test terminated satisfactorily for all the product variations and the normalized /F ratio for these variations doesn t appreciably exceed The product and its variations pass all acceptance test rehearsals planned for them 4. Related systems pass all acceptance tests far@ucalgary.ca 39

40 Example 1 TBF We want tto select an operating system as a build platform for a new software product. The Windows XP Pro operating system was evaluated for this purpose. The system log during the testing period showed 30 errors recorded in the eventlog file as shown Failure no. No. Type TBF (hours) Error Source 1 Error DCOM 2 Error Service Control Manager 3 Error 9.89 DCOM 4 Error 0.07 DCOM 5 Error 5.70 DCOM 6 Error 7.89 DCOM 7 Error DCOM 8 Error DCOM 9 Error DCOM 10 Error System Error 11 Error 7.99 DCOM 12 Error DCOM 13 Error DHCP 14 Error 1.78 System Error 15 Error DCOM 16 Error DCOM 17 Error Service Control Manager 18 Error 9.18 DCOM 19 Error DCOM 20 Error 9.19 DCOM 21 Error System Error 22 Error System Error 23 Error Service Control lmanager 24 Error 0.75 Service Control Manager 25 Error System Error 26 Error Service Control Manager 27 Error Service Control Manager 28 Error System Error 29 Error DCOM 30 Error DHCP far@ucalgary.ca 40

41 Example 1 (cont d) Calculate the system MTTF, failure intensity and reliability for 10 hours of operation, assuming that MTTF=MTBF and using exponential reliability model. 30 MTBFi i MTBF hour i failure/hour MTTF / Re 0.75 far@ucalgary.ca 41

42 Example 1 (cont d) The system must be rebooted after each System Error type error. If the recovery time to reboot the system after each System Error is 15 minutes, calculate the availability of this system. uptime Availability uptime downtime If the target failure intensity for deploying this system is 1 failure per 100 hours can we elect using this system as our build platform? λf = 0.01 and λf is smaller than λ= So the answer is no. far@ucalgary.ca 42

43 Example 2 A software system has 100, lines of code (LOC) and initial failure density ( 0 ) of 5 faults per 1,000 LOC. Initial failure intensity (λ 0 ) of the software at the start of the test is 2 failures per CPU hr. The software is released when its failure intensity objective (λ r ) has reached 6 failures per 100 CPU hrs, assuming using Musa s Basic Model. During field operation the software is not fixed. Let s assume that during field operation, failure intensity is linearly proportional to the number of faults remaining in the code; and the number of faults in the code is linearly proportional to the size of the code. far@ucalgary.ca 43

44 Example 2 (cont d) Determine the remaining number of faults in the system at the time of release. μ =5 faults/1000loc Program size is 100,000 LOC. Hence, there were initially 0 =500 faults in the program. λ 0 =2 failure/ CPU hr λ r = 6 failure/ 100 CPU hr = 0.06 failure/cpu hr Applying Musa s Basic Model: λ= λ 0 (1 μ/ 0 ) λ r = 2(1 μ r /500) =0.06 therefore μ r =485 i.e., 485 failures observed at the time of release. At the time of release, there are = 15 faults remaining in the system. far@ucalgary.ca 44

45 Example 2 (cont d) Suppose that we want to release a maintenance version of the software that includes a change of 2,000 lines of source code. Obtain the new expected failure intensity λ. In the maintenance release 2,000 LOC has been changed. There were 500 faults in 100,000 LOC therefore additional 2,000 LOC in the maintenance release will at most add an additional 10 faults in the system. Thus, there will be μ=15+10=25 faults in the system. It is assumed that failure intensity is proportional to the number of faults in the system. 15 faults 0.06 failures/cpu hr 25 faults (25/15) 006=01failures/CPUhr far@ucalgary.ca 45

46 Section 3 Test Completion & Release Criteria: Zero Failure Testing far@ucalgary.ca 46

47 Zero Failure Testing /1 How many hours the system must be tested in order to meet a quality goal? failure density Developed by Brettschnieder at Motorola Total test time n F ln 0.5 nf T ( T T ) zero 0.5 n test last F ln n T n F Time of last failure # of failures observed so far Target # of failures far@ucalgary.ca 47

48 Zero Failure Testing /2 Target projected number of failures (n F ) is calculated using target failure density Number of repairable failures observed so far (n T ) Test execution hours since the last failure found (T=T test T last ). far@ucalgary.ca 48

49 Example Suppose you are testing a 33,000 LOC program, and to date 15 repairable failures have been detected over an execution time of 500 hours. No failures have been observed in the last 50 hours of testing. Management want to know how much more testing is needed to ensure that the customers observe no more than a projected average of 0.03 failures per KLOC. far@ucalgary.ca 49

50 Example (cont d) Calculate additional testing time. n F n T ,000 1,000 1 n F ln 1 ln n F T ( ) zero Ttest Tlast (500 50) n F ln ln n 15 1 T nf Additional test hours far@ucalgary.ca 50

51 Section 4 Test Completion & Release Criteria: Special Cases far@ucalgary.ca 51

52 Special Situations What else may affect /F estimation? Program evolution Dealing with small increments of program size and/or stepwise evolution Unreported failures How unreported failures affect failure intensity estimation? Operational profile variation 52

53 Evolving Programs /1 Reliability models used in estimating i failure intensity i () assume that the executed program is stable (not changing, except for those changes that result from failure correction). Programs can evolve due to: Requirements changes Integration of new parts Necessity to adapt to changing hardware and software environment Necessity for system performance improvement Evolution as part of the development process far@ucalgary.ca 53

54 Evolving Programs /2 Possible scenarios: Ignoring changes for programs evolving slowly and in small size increments Ignoring old ldfil failure data dt of fthe old phase and base the estimates on the new phase Stepwise evolution 1. Applying changes by components 2. Applying changes by operation groups 54

55 a) Ignoring Changes The effects of evolution can be ignored for programs evolving slowly and in small size increments (e.g., increments of less than %5 of total code per week). Advantages: No extra testing ti and data collection is required. Disadvantages: Eti Estimates t of model dlparameters and /F ratio will lag the true situation and will have error, but the range of errors may be acceptable. far@ucalgary.ca 55

56 b) Ignoring Old Data Set the range of variation for the failure data to include only the recent failures. The recent failure window must usually include at least failures to estimate the parameters properly Only applicable when the changes in the program is limited (e.g., less than 20% of the programs change) 56

57 c) Stepwise Evolution /1 Stepwise evolution approaches generally work best when having a small number of large changes, each resulting from the addition of an independent element (subsystems, packages, etc.). In component by component approach, add the component failure intensities separately to obtain the system failure intensity at each stage. In operation group approach, add the weighted failure intensities of operation groups to obtain the system failure intensity at each stage. far@ucalgary.ca 57

58 c) Stepwise Evolution /2 Advantages and disadvantages d of stepwise evolution approaches: Advantages: System operational profile can be applied directly. Disadvantages: Extra data collection required because of the multiple elements Greater estimation error (or later achievement of a specified degree of accuracy) due to smaller sample sizes far@ucalgary.ca 58

59 Other Evolution Scenarios Additional situations analogous to program evolution: A program is fully integrated, but it is tested in an evolutionary fashion. One component or operation group is turned on, then another, and so on. A program is fully integrated and active, but the observation of the components for failures is handled d in an evolutionary fashion. One starts with a narrow focus of observation and then widens it component by component or operation group by operation group. far@ucalgary.ca 59

60 Unreported Failures /1 Fil Failures may not tbe reported, tdin cases such as Less severe failures Failures not halting program execution More failures may be missed in load test than in feature or regression test tbecause the exact tinputs and outputs during load test may be unknown Missing i to report failures means underestimating actual failure intensity () How unreported failures affect failure intensity estimation? far@ucalgary.ca 60

61 Unreported Failures /2 i P i P1 p2 p1 ia a P i P ia 2 Pis i the probability that the i th failure goes unnoticed Assumption: Initially, the probability of not observing a particular failure is P1 and as time goes by, this increases linearly with a constant change rate a until reaching to its final value P2 Figure from Musa s Book far@ucalgary.ca 61

62 Unreported Failures /3 For the basic model: Expected failure intensity at execution time 0 e 0 0 e The plot for The plot for ^ Figure from Musa s Book far@ucalgary.ca 62

63 Unreported Failures /4 For the basic model: For p1=p2=p p p estimates for failure intensity fluctuates around 1-p If P(i) increases with time, the estimate for is low and becomes lower as time passes If P(i) decreases with time, the estimate for is initially low but improves as time passes Correction is possible far@ucalgary.ca 63

64 Unreported Failures /5 For the logarithmic Poisson model: Expected failure intensity at execution time The plot for ^ Figure from Musa s Book far@ucalgary.ca 64

65 Unreported Failures /6 For the logarithmic Poisson model: For p1=p2=p p p estimates for failure intensity fluctuates around 1-p The estimate for is low and becomes lower as time passes Correction is possible but not as good as the basic model far@ucalgary.ca 65

66 Operational Profile Variation When a system has various operational profiles, what will be the effects of operational profiles on? D is measured failure intensity for the operational profile D and p Dk is the occurrence probability of the k th operation and Q0 is the total number of operations k is a constant for each operation calculated from the model parameters M D Q0 pmkk k 1 Q0 pdkk k 1 far@ucalgary.ca 66

67 67