Configurable Test-Goal Set Partitioning for Directed Multi- Goal Test Generation

Transcription

1 Configurable Test-Goal Set Partitioning for Directed Multi- Goal Test Generation Malte Lochau (TU Darmstadt) Joint work with: Sven Apel, Johannes Bürdek, Dirk Beyer, Andreas Holzer, Andreas Stahlbauer

2 Test-Case Generation with Software Model Checking program test goal is unreachable φφ test goal Counter example test case tttt = (II, OO)

3 Tiger Algorithm for Multi-Goal Test Coverage IIIIIIIIII: CCCCCC cccccc, SSSSSS oooo GGGGGGGGGG GG OOOOOOOOOOOO: TTTTTTTT SSSSSSSSSS TTTT cccccccccccccccc GG TTTT = {} GG GG wwwwwwwww GG gg pppppppp aaaaaa rrrrrrrrrrrr eeeeeeeeeeeeee ffffffff GG if tttt rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr cccccc, gg TTTT TTTT tttt, gg fffffffffffff gg GGG iiii tttt cccccccccccc ggg TTTT TTTT tttt, gg GGG GG \g

4 Example 1 int func-spl (int x, int y, int z) { int a; if (x < y) 4 a = x; 5 else 6 a = y; 7 z = z + a; 8 return z; 9 } iiiiii aa xx < yy xx yy

5 Example tttttttt iiiiii aa xx < yy xx yy tttttttt xx < yy 7 7 xx < yy 8 xx < yy zz 1 = zz 0 + aa 8 tc = (x=0, y=1, z=0 z=0)

6 Example tttttttt iiiiii aa xx < yy xx yy tttttttt xx < yy 6 xx yy 7 7 xx < yy 8 xx < yy zz 1 = zz 0 + aa 8

7 Example tttttttt iiiiii aa xx < yy xx yy tttttttt xx < yy 6 xx yy 7 7 xx < yy xx yy 8 xx < yy xx yy zz 1 = zz 0 + aa 8 tc =?

8 Example tttttttt iiiiii aa xx < yy xx yy tttttttt xx < yy 6 xx yy 7 7 xx < yy 7 xx yy 8 xx < yy zz 1 = zz 0 + aa 8 xx yy zz 1 = zz 0 + aa 8 tc = (x=0, y=1, z=0 z=0) tc = (x=0, y=0, z=0 z=0)

9 Example tttttttt iiiiii aa xx < yy xx yy tttttttt xx < yy 6 xx yy 7 7 xx < yy xx yy 8 xx < yy xx yy zz 1 = zz 0 + aa 8 ARG-state merge only feasible before reaching the test goal ARG is (re-)constructed for every test-goal

10 Model Checking for Test-Case Generation program φφ Counter example

11 Multi-Property Checking for Multi-Goal Test-Suite Generation program φφ φφ φφ Counter Counter example Counter example example

12 Example: Multi-Goal Test-Generation iiiiii aa xx < yy xx yy * 0 1 * * 0 1 * * 0 1 * * 0 * * 1 * Test-Goal Automata - represent sets of CFA paths - non-deterministic - parallel automata semantics: multiple simultanously active states Test-Goal Set Partitioning - Process subsets of test-goal automata simultaneously - Partitioning criteria?

13 Example: Multi-Goal Test-Generation * 0 1 * * 0 1 * * 0 1 * iiiiii aa xx < yy xx yy xx < yy tttttttt tttttttt 6 xx yy 8 7 xx < yy {0,1},{0},{0} 7 xx yy {0},{0,1},{0} 8 xx < yy zz 1 = zz 0 + aa {0,1},{0},{0,1} 8 xx yy zz 1 = zz 0 + aa {0},{0,1},{0,1}

14 ARG-State Merge Fails! * 0 1 * * 0 1 * * 0 1 * iiiiii aa xx < yy xx yy xx < yy tttttttt tttttttt 6 xx yy 8 7 xx < yy xx yy {0,1},{0,1},{0} 8 xx < yy xx yy zz 1 = zz 0 + aa {0,1},{0,1},{0,1}

15 Solution: Merge with (Test-Goal) Weaving * 0 gg 1 * * 0 1 * * gg gg * iiiiii aa xx < yy xx yy xx < yy tttttttt tttttttt 6 xx yy xx < yy aa xx = < xxyy ggaa 1 = xx1 xxxx yyyy aa yy gg = 1 {0,1},{0,1},{0} 8 xx < yy aa 8 = xxxx < gg 1 yy= aa 1 = xxxx xxyy aa yy= aa yy = ggyy = 1zz 1 = zz 10 = + aazz 0 + aa gg = 1 {0,1},{0,1},{0,1}

16 CEGAR vs. TGAR CEGAR Refinement after each covered goal iiiiii aa => basic-block coverage and beyond on larger programs!? xx < yy xx yy TGAR Refinement after k covered goals => reduce (redundant) refinement steps

17

18 Experiments Subject systems 4 programs with 1541 LoC on average Coverage criteria Basic-Block Coverage (C0) Condition Coverage (C) Test-goal set partition sizes Fixed: 1 (DS), 10 (DM10), 0 (DM0), 50 (DM50), all (DA) Percental: 5% (DS5%), 50 (DS50%), 75 (DS75%)

19 Research Questions RQ1 (Efficiency): Does multi-goal set partitioning (DM) increase testing efficiency, as compared to single-goal (DS) and all-goal (DA) processing? RQ (Effectiveness): How does multi-goal set partitioning (DM) influence testing effectiveness, as compared to single-goal (DS) and all-goal (DA) processing?

20 All-Goals (DA) vs. Single-Goal (DS) Avg. CPU time per program: 70.50s (DS) s (DA) Avg. CPU time per test goal: 1.s (DS) s (DA) Median Speedup by DA, as compared to DS: 1.4 Avg. Test-Suite Size: 4.9 (DS) (DS) Avg. Coverage Rate: 99.7% (DS) % (DA) Avg. Fault-Detection Rate: 89.9% (DS) % (DA) DA (slightly) improves testing efficiency under stable testing effectiveness, as compared to DS

21 Configurable Partition Sizes (DM) Multi-Goal improves testing efficiency and testing effectiveness, as compared to Single-Goal and All-Goal Depends on partition size We observe a critical number of test goals constituting the best trade-off between testing efficiency and testing effectiveness (i.e., goals covered per second)

22 Ongoing Work Criteria for choosing parition sizes and clusterings of test goals Static: based on CFA (static slicing ) Dynamic: on-the-fly (use information from previous CPAchecker runs )