Complex testbench design strategy

Hello,

I'm now considering the design of a rather complex testbench, and want
to consult the dwellers of this list on the best strategy to adopt.

The testbench is for a complex multi-functional FPGA. This FPGA has
several almost unrelated functions, which it makes sense to check
separately. So I'm in doubt as to how to structure my testbench. As I
see it, there are two options:

1) One big testbench for everything, and some way to sequence the
tests of the various functions
2) Separate testbenches, one for each function

Each approach has its pros and cons. Option (2) sounds the most
logical, at least at first, but when I begin pursuing it, some
problems emerge. For example, too much duplication between testbenches
- in all of them I have to instantiate the large top-level entity of
the FPGA, write the compilation scripts for all the files, and so on.
In option (1) this would have to be done only once. On the other hand,
option (2) has the clear advantage over (1) of a clean separation of
tested functionality.

I'm sure this issue haunts every verification engineer at one time or
another. What approach do you use - 1, 2 or something else, and why ?

Thanks in advance

Eli Bendersky

Eli Bendersky a écrit :
> Hello,
>
> I'm now considering the design of a rather complex testbench, and want
> to consult the dwellers of this list on the best strategy to adopt.
>
> The testbench is for a complex multi-functional FPGA. This FPGA has
> several almost unrelated functions, which it makes sense to check
> separately. So I'm in doubt as to how to structure my testbench. As I
> see it, there are two options:
>
> 1) One big testbench for everything, and some way to sequence the
> tests of the various functions
> 2) Separate testbenches, one for each function

Hello
I once had the same kind of task. I ended up with one big testbench with
a very long list of generic parameters and conditional component
instantiations.
Compilation is done once and for all, you just have to maintain the
simulation scripts that set the generic parameters for each partial
simulation.

Nicolas

Nicolas Matringe

On Nov 16, 7:50*am, Jonathan Bromley <jonathan.brom...@MYCOMPANY.com>
wrote:
> On Sun, 16 Nov 2008 03:30:34 -0800 (PST), Eli Bendersky wrote:
> >I'm now considering the design of a rather complex testbench [...]
>
> Nice description of a problem that, as you say, is common to
> almost any decent-sized verification project.
>
> Almost any advice I could offer would be more-or-less an
> echo of the sensible, practical suggestions in Janick
> Bergeron's well-known "Writing Testbenches" book. *
> I think he misses a few tricks about the best way for
> the upper levels of a TB to interact with its
> bus-functional models (BFMs), but otherwise it's
> pretty much on the money.
>
> The first edition of that book covers both VHDL and Verilog.
> The second edition adds extensive discussion of Vera and
> 'e'. *Note that there is yet a third version of the book
> which is SystemVerilog-focused; I'm guessing you don't
> want that
>
> >The testbench is for a complex multi-functional FPGA. This FPGA has
> >several almost unrelated functions, which it makes sense to check
> >separately.
>
> Yes, but most of the really "interesting" bugs are likely to
> be related to the interactions between those functions...
>
> >1) One big testbench for everything, and some way to sequence the
> >tests of the various functions
>
> I'd certainly go for that. *Better still, build a "test harness" -
> the structural guts of the TB, containing your DUT instance and
> all its life-support systems such as clock and reset generators,
> external component models, BFMs for testbench stimulus and
> monitoring, and so on. *Take very, very great care to give that
> test harness the cleanest, simplest possible interface on its
> ports. *Ideally, there should be one BFM for each major
> interface of the DUT, and each BFM should expose only two
> record ports on the TB side - one "request" port and one
> "response" port. *You don't need handshakes at that level;
> BFMs and TB can both use the 'transaction attribute on those
> record signals to detect each new request or response. *
>
> The top level of this test harness is likely to be
> horrible, with all the DUT-connection signals, DUT
> instance and a pile of other big things instantiated too.
> But you write it exactly once for the whole project.
>
> You can now instantiate this test harness into a top-level
> testbench containing only a few processes that sequence the
> activity of the various BFMs. *With luck, this top-level
> TB will have only a few hundred lines of code and will be
> fairly easy to modify to create alternative test sequences.
> If you want to exercise your blocks one at a time, you can
> easily write a top-level TB that leaves most of the BFMs
> idle and merely exercises just one of them. *Your top level
> TB can easily randomize some of its behavior if you wish;
> both the contents and the timing of the transactions (record
> values) that you send to the BFMs can be adjusted by
> creative use of ieee.math_real.uniform().
>
> The devil is in the detail, of course, but something like
> this works pretty well for me and it's what we recommend in
> our advanced VHDL verification classes.
>
> I think you can find examples of similar ideas in Jim Lewis's
> training materials too.
> --
> Jonathan Bromley, Consultant
>
> DOULOS - Developing Design Know-how
> VHDL * Verilog * SystemC * e * Perl * Tcl/Tk * Project Services
>
> Doulos Ltd., 22 Market Place, Ringwood, BH24 1AW, UK
> jonathan.brom...@MYCOMPANY.comhttp://www.MYCOMPANY.com
>
> The contents of this message may contain personal views which
> are not the views of Doulos Ltd., unless specifically stated.

I also recommend a structure similar to what Jonathan describes above.
If you haven't already used the toggling transaction trick that
Jonathan mentions, I'd recommend getting a copy of Bergeron's book and
reading it through (at least the first parts and the VHDL parts). It's
well worth it. I can count on one hand those books that I sat down
with, read through from front to back, and found that they made the
light bulb go off, creating a lasting impression, and have guided me
in the years since. This was one of those books for me.

Other benefits of a single large test harness over a bunch of
individual testbenches for submodules is that it's also easy to plug
in the gate level code for final verification of the whole FPGA if you
like. I'm not sure if by "separate testbenches, one for each function"
you were planning to expose the top-level FPGA interface or just the
sub-function interface. It would be a shame to lose the ability to
verify the whole FPGA for want of a decent test harness.

One potential drawback of the mother-of-a-test-harness-monster-code,
though, can be efficiency. Lighting up all sorts of PLL/DCM and high-
speed SERDES components in simulation can be pretty slow if you get to
gate level sims, especially if you have to light up every single one
in a large FPGA just to test one of the ten subfunctions. At the RTL
level, some very creative use of abstract high level models for, say
SERDES, can speed up runtime drastically while still letting you
validate the other 99% of the logic inside the guts of the chip.

Good luck!

- Kenn

kennheinrich@sympatico.ca

> I once had the same kind of task. I ended up with one big testbench with
> a very long list of generic parameters and conditional component
> instantiations.
> Compilation is done once and for all, you just have to maintain the
> simulation scripts that set the generic parameters for each partial
> simulation.

Can you elaborate how you setup generics in runtime for partial
simulations ? I thought generics can only be set at compile/load time
and instantiations are static at runtime. This is my biggest problem
with the monolithic testbench approach.

Eli

Eli Bendersky

On Nov 16, 2:50*pm, Jonathan Bromley <jonathan.brom...@MYCOMPANY.com>
wrote:
> On Sun, 16 Nov 2008 03:30:34 -0800 (PST), Eli Bendersky wrote:
> >I'm now considering the design of a rather complex testbench [...]
>
> Nice description of a problem that, as you say, is common to
> almost any decent-sized verification project.
>
> Almost any advice I could offer would be more-or-less an
> echo of the sensible, practical suggestions in Janick
> Bergeron's well-known "Writing Testbenches" book. *
> I think he misses a few tricks about the best way for
> the upper levels of a TB to interact with its
> bus-functional models (BFMs), but otherwise it's
> pretty much on the money.
>

Thanks - I'll definitely order it. Any other good books you can
recommend on writing testbenches ?

<snip>
> Yes, but most of the really "interesting" bugs are likely to
> be related to the interactions between those functions...
>

Yes, this is why I planned each function-testbench to instantiate the
whole top-level DUT, so as not to lose those valuable internal
interfaces. When writing TBs, we always attempt to treat the DUTs as
black boxes, without getting inside. It really helps to test the whole
path from the top level of the FPGA to internal functions.

> >1) One big testbench for everything, and some way to sequence the
> >tests of the various functions
>
> I'd certainly go for that. *Better still, build a "test harness" -
> the structural guts of the TB, containing your DUT instance and
> all its life-support systems such as clock and reset generators,
> external component models, BFMs for testbench stimulus and
> monitoring, and so on. *Take very, very great care to give that
> test harness the cleanest, simplest possible interface on its
> ports. *Ideally, there should be one BFM for each major
> interface of the DUT, and each BFM should expose only two
> record ports on the TB side - one "request" port and one
> "response" port. *You don't need handshakes at that level;
> BFMs and TB can both use the 'transaction attribute on those
> record signals to detect each new request or response. *
>
> The top level of this test harness is likely to be
> horrible, with all the DUT-connection signals, DUT
> instance and a pile of other big things instantiated too.
> But you write it exactly once for the whole project.
>
> You can now instantiate this test harness into a top-level
> testbench containing only a few processes that sequence the
> activity of the various BFMs. *With luck, this top-level
> TB will have only a few hundred lines of code and will be
> fairly easy to modify to create alternative test sequences.
> If you want to exercise your blocks one at a time, you can
> easily write a top-level TB that leaves most of the BFMs
> idle and merely exercises just one of them. *Your top level
> TB can easily randomize some of its behavior if you wish;
> both the contents and the timing of the transactions (record
> values) that you send to the BFMs can be adjusted by
> creative use of ieee.math_real.uniform().
>
<snip>

A very interesting idea! While I mused about something similar, I
couldn't think it through to a full solution on my own. Let me see if
I get you straight. You propose to 'dress' my DUT in a harness
convenient for simulation, with simple transaction-like record ports
instead of complex signal interfaces. So far I understand.
But then, for each tested function, I suppose I should have a separate
TB that instantiates this harness, right ?

This makes it more similar to method (2) in my original message, with
some of the same problems. For instance, for each of the testbenches I
will have to compile all the DUT files separately, this giving some
duplication in compile scripts. However, much of the duplication of
instantiating the DUT with its 400 pins over and over again is saved,
because the harness presents a very simple interface to the outside -
one that can be only partially instantiated.

Did I understand you correctly ?

Thanks for the advice!
Eli

Eli Bendersky

Eli Bendersky wrote:

> I'm now considering the design of a rather complex testbench, and want
> to consult the dwellers of this list on the best strategy to adopt.

Interesting question.

> This FPGA has
> several almost unrelated functions, which it makes sense to check
> separately. So I'm in doubt as to how to structure my testbench. As I
> see it, there are two options:
>
> 1) One big testbench for everything, and some way to sequence the
> tests of the various functions

I certainly have to be able to wiggle all the inputs
and watch all the outputs, so I don't see any way
to get around this one. And as long as I have
to do this anyway, it is probably a good idea
to make top testbench drill as deep as my schedule allows.

> 2) Separate testbenches, one for each function

I don't have a final answer for this one.
Design time often alternates between top down
and bottom up, and an entity/module is the
only synthesizable object that a testbench can test.
I design some modules, I reuse or buy others.
One module may be a simple synchronizer.
Another might be an entire front end.

> I'm sure this issue haunts every verification engineer at one time or
> another. What approach do you use - 1, 2 or something else, and why ?

In the something else category, i might add

1. Assertion testing of packaged functions.
If I can extract a vhdl function from a complex expression,
I can verify this much out of time by testing the package.
I also have the option on reusing these pretested functions.
I might even have some useful assertions to make about constants.

2. Avoid text files, custom languages, BFLs etc.
VHDL has functions and procedures that cover this ground.
VHDL does run out of gas at the higher level verification
abstractions, but I prefer to make due with vanilla vhdl.

3. I like the discussion in these papers.
http://enpub.fulton.asu.edu/cse517/bfm.pdf
http://www.syncad.com/paper_coding_t...u_sep_2003.htm

-- Mike Treseler

Mike Treseler

Eli Bendersky wrote:

> 1) One big testbench for everything, and some way to sequence the
> tests of the various functions

In my experience, that's the way to go. You always end up with less
duplication, and it's easier to maintain changes which might otherwise
require synchronisation between the various separate testbenches.

Another mechanism I employed - for both synthesis and simulation - is to
define a set of constants in a global package, and assign them in the
package body - that enable/disable various (unrelated) sections of the
design (using if-generate for example). You can then selectively build
various combinations of the design to simplify/speed up simulation and/or
synthesis.

In this way you could, for example, use the exact same testbench framework
but have several simulation project files with unique "global package
bodies" that build different parts of the design for each
functionally-unrelated testbench script.

Regards,

--
Mark McDougall, Engineer
Virtual Logic Pty Ltd, <http://www.vl.com.au>
21-25 King St, Rockdale, 2216
Ph: +612-9599-3255 Fax: +612-9599-3266

Mark McDougall

> 3. I like the discussion in these papers.http://enpub.fulton.asu.edu/cse517/b...u_sep_2003.htm
>

I'll look at these, thanks.

Eli Bendersky

On Nov 17, 2:42*am, Mark McDougall <ma...@vl.com.au> wrote:
> Eli Bendersky wrote:
> > 1) One big testbench for everything, and some way to sequence the
> > tests of the various functions
>
> In my experience, that's the way to go. You always end up with less
> duplication, and it's easier to maintain changes which might otherwise
> require synchronisation between the various separate testbenches.
>
> Another mechanism I employed - for both synthesis and simulation - is to
> define a set of constants in a global package, and assign them in the
> package body - that enable/disable various (unrelated) sections of the
> design (using if-generate for example). You can then selectively build
> various combinations of the design to simplify/speed up simulation and/or
> synthesis.
>
> In this way you could, for example, use the exact same testbench framework
> but have several simulation project files with unique "global package
> bodies" that build different parts of the design for each
> functionally-unrelated testbench script.
>

Sounds like an interesting approach that can save simulation/synthesis
time. One problem I see with it is that the DUT is not always under
our control. As far as I'm concerned, it's sometimes just a black box
I can assume nothing about except its spec.

Eli

Eli Bendersky

writes:

> One potential drawback of the mother-of-a-test-harness-monster-code,
> though, can be efficiency. Lighting up all sorts of PLL/DCM and high-
> speed SERDES components in simulation can be pretty slow if you get to
> gate level sims, especially if you have to light up every single one
> in a large FPGA just to test one of the ten subfunctions. At the RTL
> level, some very creative use of abstract high level models for, say
> SERDES, can speed up runtime drastically while still letting you
> validate the other 99% of the logic inside the guts of the chip.

Indeed.

I have several "layers" of TB, so I can run a restricted set of
regression tests on (say) the algorithmic part of my design, which
presents a record for its configuration to the encompassing element.
I can then sidestep the tedious setup of all the registers (which is
through SPI) and just concentrate on validating the algorithm bit.

The "right-answers" for the algorithm testbench are generated from the
Matlab environment in which the algorithm is developed. So new
algorithmic regression tests are very easily added as new situations
are discovered in "algorithm-land".

The SPI regs have their own TB to check their operation in isolation.

Large lumps of the algorithm TB are then instantiated in the higher level
testbenches, which checks that I've wired them up right etc. This can
be recycled for the post-synth and post-PAR sims which occasionally
(very occasionally in the case of the post-PAR, for power anaylsis data!)

At the top I have a python script which finds all my regression tests
and feeds them to vsim to make sure they all pass. I run this
periodically, as well as before release, but most of the time I am
working down the hierarchy, so my TBs concentrate on testing just the
lower functionality.

Cheers,
Martin

--

TRW Conekt - Consultancy in Engineering, Knowledge and Technology
http://www.conekt.net/electronics.html

Martin Thompson