As pointed out in the previous post, a not-well-verified IP can be a major killer for any SoC project. SoC integrators basically don’t have the time or resources to debug individual problematic IPs. A bad IP can really delay a project, if not kill it.

Having a robust IP signoff process is really critical and adopting reliable signoff metrics to help make signoff decisions is key. Most IP vendors provide various kinds of converge as the proof for good verification quality. However, in reality, those metrics are only telling half of the story. Most of the coverage metrics, such as functional coverage and code coverage are measuring the test distribution. Those metrics are adopted to try to make sure the test stimuli covers most of the critical functions and import corners. It assumes that whatever abnormal design behavior will be detected by the testbench. To check if the there is any exception during simulation, “checkers” inside testbench play the role of identifying the correctness. A checker can be a simple “diff” in scripts, a waveform compare utility along with the golden model, or some complicated structures.

Using coverage to qualify the verification quality is not enough. An extreme example is: a design that has 100% coverage but all checkers are disconnected by accident. Although the verification engineers still get 100% coverage, the testbench actually checks nothing. Taking the checker quality into account is definitely a must for any verification methodology, especially critical for IP signoff. Since checkers can be implemented in various forms, it is not easy to directly measure their quality. An alternative method is called mutation-testing. This is done by injecting artificial faults into a design and running the regression tests. If any introduced erroneous behavior cannot be detected by primary outputs, it implies that there might be a missing checker to check this exception behavior. The mutation-test tricks have been adopted in the software world for more than 20 years, but recently re-gained a lot of attention their application on chip design and verification. Using mutation-testing is a practical way to quantify the quality of checkers inside the testbench. It should be served as a complementary metric as any other coverage metric.

In the next blog, we will talk more about mutation-test solution and its practical usages and applications in verification. J