The “unspoken” pains of Code-cov analysis and some fresh thoughts

Earlier this week at SNUG India 2011, Intel had a very interesting and pragmatic paper on just “how rudimentary” the code coverage analysis tools are as of today. The presenter Ishwar did a great job in all aspects – right from having chosen the right number of slides, clarity and keen focus on the problem he chose to talk about.

Let’s start with the basics – below is a trivial Toggle cov report from VCS’s URG (just as a sample, every major simulator has similar viewer)

 image

While the color coding, hierarchical reporting etc. are useful – the “uncovered” hole analysis is where the users spend lot of time analyzing just “what is not covered and why”? Ishwar’s team has done some out-of-the-box thinking and shown how a simple, logical way to approach the analysis can bring quick results than the “raw” approach that many might follow – blame it on the tools not giving better assistance, if you will. He used 3 criteria to prioritize which holes to look at:

  1. How “close” is the net/node/coverage object is to the primary input?
  2. How recently did the design/block change?
  3. The fan-in/fan-out count

There was a great question from the audience on the #1 above – why not focus on “output” than the “input”. After all we are verifying the design behavior, isn’t it? A great question indeed and a matching answer from Ishwar:

Since we as “stimulus writers” have better control on the inputs, it makes it easier to “fix” the hole thereby improving the coverage number faster. After all if we didn’t even toggle the input pins fully, how do you expect the internal nodes, all the way upto output nodes would toggle?

Now add our TeamCVC’s own twist to it:

  • Identifying prioritized set of goals/candidates to look at is a critical next step to get more value out of code coverage. Clearly the industry would benefit from such simple steps especially for those teams to whom the code coverage is a MUST have sign-off criteria (most of the ASICs do demand that)
  • While in general coverage analysis is a key process in DV closure, it is important to ensure we “check” what we got “covered” too. Otherwise we get into this infamous phenomenon of “We got it covered, but didn’t get it checked”. Not always easy, especially at code coverage level, but atleast important to keep this in mind while doing the hole analysis. B’cos that’s when you realize “OK, since you told me a scenario that you didn’t cover, let me explain what would make this cover, and also what is expected behavior of the DUT in such cases, and why it is important”.
  • A novel, emerging approach to the above problem is to “Begin-with-end-in-mind”. Look at Breker’s Trek – www.brekersystems.com – the whole paradigm of “Scenario models” and “Model based test synthesis” is to start at the “expected DUT” output and expand that to “what stimulus” is needed to get there, and what shall be the expected behavior.
  • Zocalo's Zazz Bird-dog is a tool much like what Ishwar presented that does some deep analysis/heuristics to promote assertions to be written on those candidates. Now one could use the same info with code coverage analysis as well - technically speaking. This has NOT been done/endorsed yet by the developers of the tool themselves, so it is an idea from TeamCVC end based on pure technical perspective.
  • NextOp (www.nextopsoftware.com<http://www.nextopsoftware.com>) promotes a technology called "Assertion Synthesis" that infers "properties" from simulation and presents them to users/RTL designers to see if they care about these behaviors (could be classified as assertions or cover hole by the designer). Arguably, there is more work involved here by the designer but you end up high quality RTL.

Looking forward – one would hopefully expect more such papers from the real “Trenches” to highlight/guide the EDA developers to solve the key challenges from “real end-users” such as Ishwar. One topic that keeps bogging us (TeamCVC) together with our customers is the ability to do incremental “exclusion” on coverage holes – say when there are some more comments added to the source code, the old, technically/logically valid exclusion file becomes unusable/non-reusable. Any simulator expert care to comment?

Comments

Post a Comment

Popular posts from this blog

Smart one-liner for bit inversion in SystemVerilog

Recently one of CVC’s successful alumni, Harshal posted a nice challenge for SystemVerilog newcomers. Harshal has gone through our time trusted, long term SystemVerilog course and got placed at Synopsys and his career has been growing ever since. The original post describing the background is at: http://goo.gl/oq4FmF Crux of it was to “reverse the ordering” or change endianness of a bit stream. While a really rudimentary approach would be to do bit-by-bit as in:   bit [7:0] msb_vec, lsb_vec; msb_vec[7] = lsb_vec[0]; msb_vec[6] = lsb_vec[1]; // … While the above works, it is hard to maintain, upgrade for larger sizes etc. He attempted to automate it using Verilog (V2K)’s bit-slicing as in:   msb_vec [(28-i)] = lsb_vec[(0+(i))-:1]; //Bit Slicing logic But hold on, there is even a smarter way in SystemVerilog, use the “bit streaming” operator: $display (" msb_vec: %b reverse: %b", msb_vec, { << {msb_vec} } ); If the array was unppacked, there is a built-in array.reverse() ...
Read more

SystemVerilog Soft constraints usage in `uvm_do_with macro

Image
Recently we were asked a good & interesting question: How do I use "soft constraint" in the macro `uvm_do_with? What would be the syntax? I say this is a good & interesting b'cos of 2 things: 1. The SV LRM doesn't give an explicit example for this (it is fine, not that it should, LRM is not a textbook) 2. The use case should be considered (This specific user had a good need - for automatic coverage closure - or ACC). Now quickly jumping to solution, based on our SystemVerilog 2012 tutorial that our CEO Ajeetha delivered at IIT Mumbai earlier in 2014 ( http://on.fb.me/1yfcBSx ), here is a code snippet: For those who need a quick background on what are soft constraints , see: http://www.cvcblr.com/blog/?p=629   Now with 3 major EDA vendors supporting this syntax, you should leverage on this more!
Read more

Easier PLI integration with MPSim

Image
Several engineers working and aspiring to work in the field of ASIC front-end design/verification tend to stay away from Verilog’s powerful PLI/VPI – some of them because they see it as old technology and many others – simply “FEAR” from it, thanks to its complex integration with the tools. For those who believe it is old technology – think again, most of the cutting edge EDA innovations (in front end) happening around verification use Verilog’s PLI/VPI to talk to your underlying simulator. To quote a few examples: NextOp /Atrenta’s BugScope   Novas’s Verdi/Debussy Axiom’s @Designer http://www.axiom-da.com (Debugger that can work with all standard Verilog simulators) Trek from Brekersystems ( www.brekersystems.com ) OnPoint from vennsa-da.com Now for those who “fear” from VPI due to its integration challenges – to be fair – you’ve reasons to do so. However advanced functional verification solutions such as VCS, MPSim ( http://www.axiom-da.com ) provide a...
Read more