SoC Verification Methodology

SoC Verification MethodologyProf. Chien-Nan LiuTEL: 03-4227151 ext:4534Email: jimmy@ee.ncu.edu.tw1

Outlinel Verification Overviewl Verification Strategiesl Tools for Verificationl SoC Verification Flow2

Verification Problemsl Was the spec correct ?l Did the design team understand the spec?l Was the blocks implemented correctly?l Were the interfaces between the blockscorrect?l Does it implement the desiredfunctionality?l ……4

Verification Complexityl For a single flip-flop:– Number of states = 2– Number of test patterns required = 4l For a Z80 microprocessor (~5K gates)– Has 208 register bits and 13 primary inputs– Possible state transitions = 2 bits+inputs = 2 221– At 1M IPS would take 10 53 years to simulate alltransitionsl For a chip with 20M gates– ??????*IPS = Instruction Per Second8

When is Verification Complete ?l Some answers from real designers:– When we run out of time or money– When we need to ship the product– When we have exercised each line of the HDL code– When we have tested for a week and not found anew bug– We have no idea!!l Designs are often too complex to ensure fullfunctional coverage– The number of possible vectors greatly exceeds thetime available for test9

Error-Free Design ?l As the number of errors left to be founddecreases, the time and cost to identifythem increasesl Verification can only show the presenceof errors, not their absencel Two important questions to be solved:– How much is enough?– When will it be done?11

Reference Bookl System-on-a-Chip VerificationMethodology and TechniquesllbyPrakash RashinkarPeter PatersonLeena SinghCadence Design Systems Inc., USApublished byKluwer Academic Publishers, 200112

Outlinel Verification Overviewl Verification Strategiesl Tools for Verificationl SoC Verification Flow13

Verification Approachesl Top-down verification approach– From system to individual componentsl Bottom-up verification approach– From individual components to systeml Platform-based verification approach– Verify the developed IPs in an existing platforml System interface-based verification approach– Model each block at the interface level– Suitable for final integration verification14

Advs. of Bottom-up Approachl Localityl Catching bugs is easier and faster withfoundational IPs (sub-blocks)l Design the SoC chip with these highlyconfidence “bug-free” IPs15

Verification EnvironmentTestbenchInput Pattern(Stimulus)DesignunderVerificationDUVOutput(Response)16

Terminologyl Verification environment– Commonly referred as testbench (environment)l Definition of a testbench– A verification environment containing a set ofcomponents [such as bus functional models (BFMs),bus monitors, memory modules] and theinterconnect of such components with the designunder-verification(DUV)l Verification (test) suites (stimuli, patterns,vectors)– Test signals and the expected response under giventestbenches17

Testbench Designl Auto or semi-auto stimulus generator is preferredl Automatic response checking is highlyrecommendedl May be designed with the following techniques– Testbench in HDL– Tesebench in programming language interface (PLI)– Waveform-based– Transaction-based– Specification-based18

Types of Verification Tests (1/2)l Random testing– Try to create scenarios that engineers donot anticipatel Functional testing– User-provided functional patternsl Compliances testingl Corner case testingl Real code testing (application SW)– Avoid misunderstanding the spec.19

Types of Verification Tests (2/2)l Regression testing– Ensure that fixing a bug will not introduceanother bug(s)– Regression test system should be automated• Add new tests• Check results and generate report• Distribute simulation over multiple computer– Time-consuming process when verificationsuites become large20

Bug Trackingl A central database collecting knownbugs and fixesl Avoid debugging the same bug multipletimesl Good bug report system helpsknowledge accumulation21

Bug Rate Trackingl Bug rate usually follow a well-definedcurvel The position on the curve decides themost-effective verification approachl Help determine whether the SoC isready to tape-out22

Bug Rate Tracking Example# of bugs reported Time23

Adversarial Testingl For original designers– Focus on proving the design works correctlyl Separate verification team– Focus on trying to prove the design is broken– Keep up with the latest tools andmethodologiesl The balanced combination of these twogives the best results24

Verification Planl Verification plan is a part of the design reportsl Contents– Test strategy for both blocks and top-level module– Testbench components – BFM, bus monitors, …...– Required verification tools and flows– Simulation environment including block diagram– Key features needed to be verified in both levels– Regression test environment and procedure– Clear criteria to determine whether the verificationis successfully complete25

Benefits of Verification Planl Verification plan enables– Developing the testbench environment early– Developing the test suites early– Developing the verification environment in parallelwith the design task by a separate team– Focusing the verification effort to meet theproduct shipment criteria– Forcing designers to think through the timeconsumingactivities before performing them26

Outlinel Verification Overviewl Verification Strategiesl Tools for Verificationl SoC Verification Flow27

Tools for Verification (1/4)l Simulation– Event-driven:• Timing accurate– Cycle-based:• Faster simulation time (5x – 100x)– Transaction-based:• Require bus functional model (BFM) of eachdesign• Only check the transactions between components• Have faster speed by raising the level ofabstraction28

Event-Driven Simulationl Timing accuratel Good debugging environmentl Simulation speed is slowerScheduleNewEventsAdvanceTimeMore eventsfor this timeYGetCurrentEventEventEvaluationPropagateEventsNNTime wheelemptyYFinish29

Cycle-Based Simulationl Perform evaluations justbefore the clock edge– Repeatedly triggered eventsare evaluated only oncein a clock cyclel Faster simulation time (5x – 100x)l Only works for synchronous designsl Only cycle-accurateClockInputOutputl Require other tools (ex: STA) to check timingproblems30

Simulation-Based VerificationlllllStill the primary approach for functional verification– In both gate-level and register-transfer level (RTL)Test cases– User-provided (often)– Randomly generatedHard to gauge how well a design has been tested– Often results in a huge test bench to test large designsNear-term improvements– Faster simulators• Compiled code, cycle-based, emulation, …– Testbench tools• Make the generation of pseudo-random patterns better/easierIncremental improvements won’t be enough31

Tools for Verification (2/4)l Code coverage– Qualify a particular test suite when applied to aspecific design– Verification Navigator, CoverMeter, …l Testbench (TB) automation– A platform (language) providing powerful constructsfor generating stimulus and checking response– VERA, Specman Elite, …l Analog/mixed-signal (AMS) simulation– Build behavior model of analog circuits for systemsimulation– Verilog-A, VHDL-A, …32

Coverage-Driven Verificationl Coverage reports can indicate how much ofthe design has been exercised– Point out what areas need additional verificationl Optimize regression suite runs– Redundancy removal (to minimize the test suites)– Minimizes the use of simulation resourcesl Quantitative sign-off (the end of verificationprocess) criterionl Verify more but simulate less33

The Rate of Bug Detectionsource : “Verification Methodology Manual For Code Coverage In HDL Designs” by Dempster and Stuart34

Coverage AnalysisllllDedicated tools are required besides the simulatorSeveral commercial tools for measuring Verilog andVHDL code coverage are available– VCS (Synopsys)– NC-Sim (Cadence)– Verification navigator (TransEDA)Basic idea is to monitor the actions during simulationRequire supports from the simulator– PLI (programming language interface)– VCD (value change dump) files35

Analysis Results• Verification Navigator (TransEDA)Untested code line will be highlighted36

Testbench Automationl Require both generator and predictor in anintegrated environmentl Generator: constrained random patterns– Ex: keep A in [10 … 100]; keep A + B == 120;– Pure random data is useless– Variations can be directed by weighting options– Ex: 60% fetch, 30% data read, 10% writel Predictor: generate the estimated outputs– Require a behavioral model of the system– Not designed by same designers to avoidcontaining the same errors37

Analog Behavioral Modelingl A mathematical model written in HardwareDescription Languagel Emulate circuit block functionality by sensingand responding to circuit conditionsl Available Analog/Mixed-Signal HDL:– Verilog-A– VHDL-A– Verilog-AMS– VHDL-AMS38

Mixed Signal SimulationlAvailable simulators:CadenceAntrimMentorSynopsys……39

Tools for Verification (3/4)l Static technologies– Lint checking:• A static check of the design code• Identify simple errors in the early design cycle– Static timing analysis:• Check timing-related problems without input patterns• Faster and more complete if applicable– Formal verification:• Check functionality only• Theoretically promise 100% coverage but designsize is often limited due to high resource requirement40

HDL Linterl Fast static RTL code checker– Preprocessor of the synthesizer– RTL purification (RTL DRC)• Syntax, semantics, simulation– Check for built-in or user-specified rules• Testability checks• Reusability checks• ……– Shorten design cycle• Avoid error code that increases design iterations41

Inspectionl For designers, finding bugs by carefulinspection is often faster then that by simulationl Inspection process– Design (specification, architecture) review– Code (implementation) review• Line-by-line fashion• At the sub-block levell Lint-liked tools can help spot defects withoutsimulation– Nova ExploreRTL, VN-Check, ProVerilog, …42

What is STA ?lllSTA = static timing analysisSTA is a method for determining if a circuit meetstiming constraints without having to simulateNo input patterns are required– 100% coverage if applicableADQDQZFF1FF2CLKReference :Synopsys43

Formal Verificationl Ensure the consistency with specification forall possible inputs (100% coverage)l Primary applications– Equivalence Checking– Model Checkingl Solve the completeness problem in simulationbasedmethodsl Cannot handle large designs due to its highcomplexityl Valuable, but not a general solution44

Equivalence CheckingRTL or NetlistSynthesisEquivalenceCheckingRTL or NetlistlllGate-level to gate-level– Ensure that some netlist post-processing did not change thefunctionality of the circuitRTL to gate-level– Verify that the netlist correctly implements the original RTL codeRTL to RTL– Verify that two RTL descriptions are logically identical45

Equivalence Checkingl Compare two descriptions to check theirequivalencel Gaining acceptance in practice– Abstract, Avant!, Cadence, Synopsys, Verplex,Verysys, …l But the hard bugs are usually in bothdescriptionsl Target implementation errors, not design errors– Similar to check C v.s. assembly language46

Model CheckingRTL CodingSpecificationRTLInterpretationAssertionsModelCheckingl Formally prove or disprove some assertions(properties) of the designl The assertions of the design should be providedby users first– Described using a formal language47

Model Checkingl Enumerates all states in the STG of thedesign to check those assertionsl Gaining acceptance, but not widely used– Abstract, Chrysalis, IBM, Lucent, Verplex,Verysys, …l Barrier: low capacity (~100 register bits)– Require extraction (of FSM controllers) orabstraction (of the design)– Both tend to cause costly false errors48

New Approachesl The two primary verification methods both havetheir drawbacks and limitations– Simulation: time consuming– Formal verification: memory explosionl We need alternate approaches to fill theproductivity gap– Verification bottleneck is getting worsel Semi-formal approaches may be the solution– Combine the two existing approaches– Completeness (formal) + lower complexity (simulation)49

Tools for Verification (4/4)l Hardware modeling– Pre-developed simulation models for otherhardware in a system– Smart Model (Synopsys)l Emulation– Test the system in a hardware-liked fashionl Rapid prototyping– FPGA– ASIC test chip50

Assistant Hardwarel Rule of thumb– 10 cycles/s for software simulator– 1K cycles/s for hardware accelerator– 100K cycles/s for ASIC emulatorl Hardware accelerator– To speed up logic simulation by mappinggate level netlist into specific hardware– Examples: IKOS, Axis, ….51

Emulationl Emulation– Verify designs using real hardware (like FPGA?)– Better throughput in handling complex designs– Inputs should be the gate-level netlist– Synthesizable testbenches required– Require expensive emulators– Software-driven verification• Verify HW using SW• Verify SW using HW– Interfaced with real HW components– Examples: Aptix, Quicktum, Mentor’s AVS ….52

Prototypingl FPGA– Performance degradation– Limited design capacity (utilization)– Partitioning and routing problemsl Emulation system– FPGA + Logic modeler– Automatic partitioning and routing underEDA SW control– More expensive53

Limited Productionl Even after robust verification process andprototyping, it’s still not guaranteed to bebug-freel Engineering samplesl A limited production for new macro isnecessary– 1 to 4 customers– Small volume– Reducing the risk of supporting problemsl Same as real cases but more expensive54

Comparing Verification OptionsSource : “System-on-a-chip Verification – Methodology and Techniques” by P. Rashinkar, etc., KAP, 2001.55

Outlinel Verification Overviewl Verification Strategiesl Tools for Verificationl SoC Verification Flow56

SOC Verification MethodologySource : “System-on-a-chip Verification – Methodology and Techniques” by P. Rashinkar, etc., KAP, 2001.57

System Verification Stepsl Verify the leaf IPsl Verify the interface among IPsl Run a set of complex applicationsl Prototype the full chip and run theapplication softwarel Decide when to release for massproduction58

Interesting Observationsl 90% of ASICs work at the first silicon but only50% work in the target system– Do not perform system level verification (withsoftware)l If a SoC design consisting of 10 blocks– P(work)= .9 10 =.35l If a SoC design consisting of 2 new blocksand 8 pre-verified robust blocks– P(work) = .9 2 * .98 8 =.69l To achieve 90% of first-silicon success SoC– P(work) = .99 10 =.9059

Interface Verificationl Inter-block interfaces– Point-to-point– On-chip bus (OCB)– Simplified models are used• BFM, bus monitors, bus checkers60

Check System Functionality (1/2)l Verify the whole system by using fullfunctional models– Test the system as it will be used in thereal worldl Running real application codes (such asboot OS) for higher design confidence– RTL simulation is not fast enough toexecute real applications61

Check System Functionality (2/2)l Solutions– Move to a higher level of abstraction forsystem functional verification– Formal verification– Use assistant hardware:• Hardware accelerator• ASIC emulator• Rapid-prototyping(FPGA)• Logic modeler• …62

HW/SW Co-Simulationl Couple a software execution environment witha hardware simulatorl Simulate the system at higher levels– Software normally executed on an Instruction SetSimulator (ISS)– A Bus Interface Model (BIM) converts softwareoperations into detailed pin operationsl Allows two engineering groups to talk togetherl Allows earlier integrationl Provide a significant performance improvementfor system verification– Have gained more and more popularity63

System-Level TestbenchlllAll functionality stated in thespec. should be coveredThe success and failureconditions must be definedfor each testPay particular attention to:– Corner cases– Boundary conditions– Design discontinuities– Error conditions– Exception handlingSource : “System-on-a-chip Verification – Methodology and Techniques” by P. Rashinkar, etc., KAP, 2001.64

Timing Verificationl STA (static timing analysis) is the fastestapproach for synchronous designs– Avoid any timing exceptions is extremely importantl Gate-level simulation (dynamic timing analysis)– Most useful in verifying timing of asynchronouslogic, multi-cycle paths and false paths– Much slower run-time performance– Gate-level sign-off65

Design Sign-offllllSign-off is the final step in thedesign processIt determines whether the design isready to be taped out for fabricationNo corrections can be made afterthis stepThe design team needs to beconfident that the design is 100%correct– Many items need to be checkedSource : “System-on-a-chip Verification – Methodology and Techniques” by P. Rashinkar, etc., KAP, 2001.66

Traditional Sign-offl Traditional sign-off simulation– Gate level simulation with precise timing under agiven parallel verification vectors– Verify functionality and timing at the same time(dynamic timing analysis, DTA)– Parallel verification vectors also serve as themanufacturing test patternsl Problems– Infeasible for million gates design (take too muchsimulation time)– Fault coverage is low (60% typically)– Critical timing paths may not be exercised67

SoC Sign-off Schemel Formal verification used to verifyfunctionalityl STA for timing verificationl Gate level simulation– Supplement for formal verification and STAl Full scan BIST for manufacturing test68