Assertion-Based Verification

1 Assertion-Based Verification Harry D. Foster Chief Scientist Verification IC Verification Solutions Division February 2018 Outline How Verification is Done Today What Makes Verification Difficult Observability and Controllability Challenge Assertion-Based Verification Industry Case Studies Conclusions 2 H Foster, EE 382M-11, 2 HF, UT Austin, Spring 2018

2 HOW VERIFICATION IS DONE TODAY What is Verification? Verification is a process of ensuring that a design implementation meets its specification. 4 H Foster, EE 382M, 4 HF, UT Austin, Spring 2018

3 Simulation-Based Techniques Fundamental verification technique in use today Generally scales well Testing all possible states is generally incomplete Measure Coverage Simulation Testbench Generate Stimulus Design Model Check Results Assertions can be used to check results and measure coverage 5 H Foster, EE 382M, 5 HF, UT Austin, Spring 2018 Simulation Traversal Through the State Space // SystemVerilog Assertion initial state property p_comp; @(posedge clk) E -> (A==B); endproperty assert property (p_comp); 6 H Foster, EE 382M, 6 HF, UT Austin, Spring 2018

4 Time Explosion Problem How long would it take to exhaustively simulate this example? 1000000011101011011011110111 A [31:0] 101010001000110101110100101 B [31:0] E // SystemVerilog Assertion property p_comp; @(posedge clk) E -> (A==B); endproperty assert property (p_comp); 2 64 vectors X 1 vector every micro-second = 584,941 years An extremely fast simulator by today s standards! 7 H Foster, EE 382M, 7 HF, UT Austin, Spring 2018 Simulation and the Time Explosion Problem 2 64 vectors X 1 vector every micro-second = 584,941 years 8 H Foster, EE 382M, 8 HF, UT Austin, Spring 2018

5 Formal-Based Techniques Does not require a testbench or input stimulus! Automatically uses algorithms to verify the functionality Verification can be complete Complements simulation-based techniques Formal Tool Pass? Yes Done No Design Model Assertions 9 H Foster, EE 382M, 9 HF, UT Austin, Spring 2018 Conceptual Formal Tool a T x x y T x (a,x,y) // next state z 10 H Foster, EE 382M, 10 HF, UT Austin, Spring 2018

6 How is formal different than simulation? initial states // SystemVerilog Assertion property p_comp; @(posedge clk) E -> (A==B); endproperty assert property (p_comp); Very fast! 11 H Foster, EE 382M, 11 HF, UT Austin, Spring 2018 State Space Explosion There are more states in today s design than there are atoms in the universe! How many states exist in a typical design today? 12 H Foster, EE 382M, 12 HF, UT Austin, Spring 2018

7 WHAT MAKES VERIFICATION DIFFICULT INDUSTRY DRIVERS Rising Design Complexity

8 Rise in the Average Number of IP Blocks Avergage Number of IP Blocks 90 80 70 60 50 40 30 20 10 0 Closing the Design Productivity Gap! 2007 2008 2009 2010 2011 2012 2013 2014 2015* 2016* 2017* 2018* Avg. Number of 'Other' SIP Blocks Avg. Number of CPU / DSP / Controllers Avg. Number of Embedded Memory Blocks Source: Semico Research Corp. 15 H Foster, EE 382M, 15 HF, UT Austin, Spring 2018 Design Engineers are Being Productive 1,000,000,000,000,000 100,000,000,000,000 Transistors produced per electronic engineer more than 5-orders of magnitude since 1985 10,000,000,000,000 1,000,000,000,000 100,000,000,000 Quantity 10,000,000,000 1,000,000,000 100,000,000 10,000,000 1,000,000 100,000 10,000 Transistors Produced Total Electronic Engineers Source: Technology Research Group EDA Database, 1986, EDA TAM, 1989 & Gartner/Dataquest 2005 Seat Count Report, Gary Smith EDA, 2013 Seat Count Analysis, VLSI Research, 2013 - Transistors Produced Analysis 16 H Foster, EE 382M, 16 HF, UT Austin, Spring 2018

9 Design Engineers are Being Productive! Revenue/Transistor ($) 1.E+03 1.E+02 1.E+01 1.E+00 1.E-01 1.E-02 1.E-03 1.E-04 1.E-05 1.E-06 1.E-07 1.E-08 Semiconductor Learning Curve 1954 2012 Adjusted for Inflation Growth of Transistor Volume Leads to Sustained ~ 30% per Year Cost Reduction 1.E-09 1.E-10 1.0E+04 1.0E+06 1.0E+08 1.0E+10 1.0E+12 1.0E+14 1.0E+16 1.0E+18 1.0E+20 Source: VLSI Research, SIA, Federal Reserve Note: Revenue adjusted for Inflation 1954-2012 Cumulative Transistors Shipped 17 H Foster, EE 382M, 17 HF, UT Austin, Spring 2018 We are Keeping Up with Design Complexity! Thanks to Automation and Reuse! 18 H Foster, EE 382M, 18 HF, UT Austin, Spring 2018

10 Another View of Moore s Law EDA Cost per Transistor and Total IC Revenue per Transistor Both Decrease About 30% per Year 1.00E-04 1.00E-04 EDA Cost/Transistor ($) 1.00E-05 1.00E-06 1.00E-07 1.00E-08 1.00E-05 1.00E-06 1.00E-07 1.00E-08 IC Revenue/Transistor ($) 1.00E-09 1.00E-09 1.00E+13 1.00E+14 1.00E+15 1.00E+16 1.00E+17 1.00E+18 1.00E+19 1.00E+20 Cumulative Transistors Shipped Note: EDA Cost Consists of EDA License and Maintenance revenue adjusted for Inflation Source: SIA, VLSI Research, Federal Reserve EDA Cost/transistor IC Revenue/transistor 19 H Foster, EE 382M, 19 HF, UT Austin, Spring 2018 Demand for Design Engineers Grows Slowly 12 CAGR Designers 3.6% ASIC/IC Mean Peak Number of Engineers 10 8 6 4 2 7.80 8.10 8.53 10.05 10.48 Design Engineers 0 2007 2010 2012 2014 2016 20 H Foster, EE 382M, 20 HF, UT Austin, Spring 2018

11 But what about Verification Productivity? ASIC/IC Mean Peak Number of Engineers 12 10 8 6 4 2 CAGR Designers 3.6% CAGR Verifiers 10.4% 7.80 4.8 8.10 7.6 8.53 8.4 10.05 11.0 10.48 11.6 Design Engineers Verification Engineers 0 2007 2010 2012 2014 2016 21 H Foster, EE 382M, 21 HF, UT Austin, Spring 2018 INDUSTRY DRIVERS Rising Verification Complexity

12 The Emergence of New Layers of Verification Software Verification Layers Security Domains Power Domains Clock Domains Functional 23 H Foster, EE 382M, 23 HF, UT Austin, Spring 2018 What Makes Verification Difficult? Channel Encoder Decoder TX Data Link Layer PHY Single, sequential data streams Floating point unit Graphics shading unit DSP convolution unit MPEG decode... Compressed Audio RX Multiple, concurrent data streams Cross bar Bus traffic controller DMA controller Standard I/F (e.g., PCIe)... Sequential data streams 1x number of bugs -Ted Scardamalia, internal IBM study Concurrent data streams 5x number of bugs 24 H Foster, EE 382M, 24 HF, UT Austin, Spring 2018

13 Directed-Test Approach Imagine verifying a car using a directed-test approach Requirement: Fuse will not blow under any normal operation Scenario 1: accelerate to 37 mph, pop in the new Lady GaGa CD, and turn on the windshield wipers 25 H Foster, EE 382M, 25 HF, UT Austin, Spring 2018 A FEW WEEKS LATER

14 Directed-Test Approach Imagine verifying a car using a directed-test approach Requirement: Fuse will not blow under any normal operation Scenario 714: accelerate to 48 mph, roll down the window, and turn on the left-turn signal 27 H Foster, EE 382M, 27 HF, UT Austin, Spring 2018 The Concurrency Challenge A purely directed-test methodology does not scale Imagine writing a directed test for this scenario! Truly heroic effort but not practical 28 H Foster, EE 382M, 28 HF, UT Austin, Spring 2018

15 Finding Corner Case Bugs Due to Concurrency Directed-test-based simulation finds the bugs you can think of Constrained-random simulation finds the bugs you never anticipated! 29 H Foster, EE 382M, 29 HF, UT Austin, Spring 2018 Concurrency is Complicated to Verify Packet-Based Design From Fabric Tx Transaction Layer Packet Reformater Retry Buffer Arbiter To PHY Data Link Layer Packet Reformater Rx From Rx Channel 30 H Foster, EE 382M, 30 HF, UT Austin, Spring 2018

16 Adoption Trends in Verification Techniques Code coverage Assertions 2007 2012 2014 Functional coverage Constrained-Random Simulation 0% 10% 20% 30% 40% 50% 60% 70% 80% Design Projects Source: Wilson Research Group and Mentor Graphics, 2014 Functional Verification Study 31 H Foster, EE 382M, 31 HF, UT Austin, Spring 2018 OBSERVABILITY & CONTROLLABILITY

17 Fundamental Challenge of Verification DUT 0111010100111010101010000000001110101101101111011 1. Activate Stimulus A A A 3. Detect A 2. Propagate Checkers A = Assertions 33 H Foster, EE 382M, 33 HF, UT Austin, Spring 2018 Observability vs. Controllability Test didn t set up the condition to propagate the bug bug A 0 1 0 0 1 1 Assertions improve observability and reduce the need to propagate bugs 34 H Foster, EE 382M, 34 HF, UT Austin, Spring 2018

18 Poor Observability Misses Bugs Code coverage measures controllability 100% code coverage does not mean all bugs are detected [S. Devadas, A. Ghosh, and K. Keutzer. DAC 1996] DAC paper study found cases where: Code Coverage Achieved % of covered lines observable 90% Covered Only 54% Observable 100% Covered Only 70% Observable 35 H Foster, EE 382M, 35 HF, UT Austin, Spring 2018 Assertions Improve Observability Testbench = Bugs missed due to poor observability = Reduce debugging up to 50% [CAV 2000, IBM FoCs paper] Bugs detected closer to their source due to improved observability 36 H Foster, EE 382M, 36 HF, UT Austin, Spring 2018

19 2014 Where Verification Engineers Spend Their Time 37% Test Planning 24% 3% 14% Testbench Development Creating Test and Running Simulation Debug 22% Other Source: Wilson Research Group and Mentor Graphics, 2014 Functional Verification Study 37 H Foster, EE 382M, 37 HF, UT Austin, Spring 2018 Designers Spend a Lot of Time in Verification & Debug 60% Mean time design engineer spends in design vs. verification 55% 50% 54% 46% 51% 49% 53% Doing Design Doing Verification 53% 47% 45% 47% 40% 2007 2010 2012 2014 Source: Wilson Research Group and Mentor Graphics, 2014 Functional Verification Study 38 H Foster, EE 382M, 38 HF, UT Austin, Spring 2018

20 ASSERTION-BASED VERIFICATION Assertion-Based Verification How can one check a large routine in the sense of making sure that it s right? In order that the man who checks may not have too difficult a task, the programmer should make a number of definite assertions which can be checked individually, and from which the correctness of the whole program easily flows. Alan Turing, 1949 40 H Foster, EE 382M, 40 HF, UT Austin, Spring 2018

21 Property Property a statement of design intent used to specify behavior test env Testbench DUT 41 H Foster, EE 382M, 41 HF, UT Austin, Spring 2018 Assertion Property a statement of design intent used to specify behavior Assertion A verification directive test env Testbench A Trace from simulation DUT 42 H Foster, EE 382M, 42 HF, UT Austin, Spring 2018

22 High-Level Assertion Property a statement of design intent used to specify behavior Assertion A verification directive High-level Architectural focused Can be part of testbench test env Testbench A Trace from simulation DUT 43 H Foster, EE 382M, 43 HF, UT Austin, Spring 2018 Low-Level Assertion Property a statement of design intent used to specify behavior Assertion A verification directive High-level Architectural focused Can be part of testbench RTL Low-level Implementation focused Embedded in or bind to the RTL // Assert that the FIFO controller // cannot overflow nor underflow A A 44 H Foster, EE 382M, 44 HF, UT Austin, Spring 2018

23 How Assertions Are Used Today State Search Testbench RTL Formal Prop s passing tests Assertions improved bug rate FPGA or Emulation Formal Verification Simulation O/S Trials [Foster, Larsen, Turpin - DVCon 2006] 45 H Foster, EE 382M, 45 HF, UT Austin, Spring 2018 Who should create the assertions? Verification Engineer Design Engineer High-Level Assertions Requirement focused Black-box assertions Accounted for in testplan Compliance traceability Create reusable ABV IP Low-Level Assertions Implementation focused White-box assertions Not accounted for in testplan Improve observability Reduce debugging time 46 H Foster, EE 382M, 46 HF, UT Austin, Spring 2018

24 Who should create high-level assertions? Verification Engineer Design Engineer High-Level Assertions Requirement focused Black-box assertions Accounted for in testplan Compliance traceability Create reusable ABV IP Low-Level Assertions Implementation focused White-box assertions Not accounted for in testplan Improve observability Reduce debugging time 47 H Foster, EE 382M, 47 HF, UT Austin, Spring 2018 Who should create low-level assertions? Verification Engineer Design Engineer High-Level Assertions Requirement focused Black-box assertions Accounted for in testplan Compliance traceability Create reusable ABV IP Low-Level Assertions Implementation focused White-box assertions Not accounted for in testplan Improve observability Reduce debugging time 48 H Foster, EE 382M, 48 HF, UT Austin, Spring 2018

25 Specifying Design Intent Assertions allow us to specify design intent in a way that lends itself to automation clk reset_n req0 req1 Arbiter grant0 grant1 // Assert that the grants for our simple arbiter are mutually exclusive 49 H Foster, EE 382M, 49 HF, UT Austin, Spring 2018 Identifying the Error Condition For our arbiter example, we can write a Boolean expression for the error condition, as follows: clk reset_n req0 req1 Arbiter grant0 grant1 (grant0 & grant1) // error condition 50 H Foster, EE 382M, 50 HF, UT Austin, Spring 2018

26 Checking the Error Condition before Assertions Doesn t lend itself to automation. module arbiter (clk, rst_n, req0, req1, grant0, grant1);... always @(posedge clk or negedge rst_n) begin if (rst_n!= 1 b0) // active low reset if (grant0 & grant1) $display ( ERROR: Grants not mutex );... endmodule Error Condition Boolean Expression 51 51 H Foster, EE 382M, HF, UT Austin, Spring 2018 Assertion Language Adoption Design Projects 80% 70% 60% 50% 40% 30% 2007 World 2012 World 2014 World 20% 10% 0% Accellera Open Verification Library (OVL) SystemVerilog Assertions (SVA) PSL Assertion Languages and Libraries Other * Multiple answers possible Source: Wilson Research Group and Mentor Graphics, 2014 Functional Verification Study 52 H Foster, EE 382M, 52 HF, UT Austin, Spring 2018

27 IEEE 1800 SystemVerilog Mutex Example grant0 and grant1 must be mutually exclusive clk grant0 grant1 error assert property ( @(posedge clk) disable iff (~rst_n)!(grant0 & grant1)); 53 H Foster, EE 382M, 53 HF, UT Austin, Spring 2018 IEEE 1850 PSL Mutex Example grant0 and grant1 must be mutually exclusive clk grant0 grant1 error assert always (!(grant0 & grant1) abort ~rst_n) @(posedge clk); 54 H Foster, EE 382M, 54 HF, UT Austin, Spring 2018

28 Accellera OVL Mutex Example grant0 and grant1 must be mutually exclusive clk grant0 grant1 error ovl_never a_mutex (clk, rst_n, (grant0 & grant1)); 55 H Foster, EE 382M, 55 HF, UT Austin, Spring 2018 INDUSTRY CASE STUDIES

29 Published Data on Assertions Use Percentage bugs found by various techniques Assertion Monitors 34% Cache Coherency Checkers 9% Register File Trace Compare 8% Memory State Compare 7% End-of-Run State Compare 6% PC Trace Compare 4% Self-Checking Test 11% Simulation Output Inspection 7% Simulation Hang 6% Other 8% Kantrowitz and Noack [DAC 1996] Assertion Monitors 25% Register Miscompare 22% Simulation "No Progress 15% PC Miscompare 14% Memory State Miscompare 8% Manual Inspection 6% Self-Checking Test 5% Cache Coherency Check 3% SAVES Check 2% Taylor et al. [DAC 1998] 17% of bugs found by assertions on Cyrix M3(p1) project [Krolnik '98] 50% of bugs found by assertions on Cyrix M3(p2) project [Krolnik 98] 85% of bugs found using over 4000 assertions on an HP server chipset project [Foster and Coelho HDLCon 2001] Thousands of assertions in Intel Pentium project [Bentley 2001] 10,000 OVL assertion in Cisco project [Sean Smith 2002] 57 H Foster, EE 382M, 57 HF, UT Austin, Spring 2018 DAC 2008 Sun paper with lots of metrics Assertion-Based Verification of a 32 thread SPARC CMT Processor [Turumella, Sharma, DAC 2008] Category Unique Instantiated Low-Level 3912 132773 Interface 5004 44756 High-Level 1930 18618 Bugs Found by Type of Assertion Low-level Interface High-level 58 H Foster, EE 382M, 58 HF, UT Austin, Spring 2018

30 Significant reduction in debugging time Assertion-Based Verification of a 32 thread SPARC CMT Processor [Turumella, Sharma, DAC 2008] Category Unique Instantiated Low-Level 3912 132773 Interface 5004 44756 High-Level 1930 18618 Hours 16 14 12 10 8 6 4 2 0 Average Debug Time >50% 85% Formal Sim + Assert Sim + None Formal Sim + Assert Sim + None 59 H Foster, EE 382M, 59 HF, UT Austin, Spring 2018 SUMMARY

31 Assertion-Based Verification The process of creating assertions forces the engineer to think... and in this incredible world of automation, there is no substitute for thinking. 61 H Foster, EE 382M, 61 HF, UT Austin, Spring 2018 www.mentor.com