Fog Creek Software
Discussion Board

Testing Taking Too Long

It takes a long time to test the systems that I work on. (I work in ASIC hardware design, which is basically software development.)

Running a modular regression batch takes about one day, and running a system level regression batch takes about a week. (It's run over a distributed simulation farm.)

As a result, it's impossible to fully test any changes before commiting them to source control.

Given this kind of restriction, how would you deal with testing?

Over here, it's just a big chaos. Mostly people just release code without testing at all.

Any tips/hints?

Wednesday, September 01, 2004

It's really the same issue as system testing large
software projects.

* test components in isolation as much as possible
* see if you can speed up the testing by having a larger farm, bigger iron, or faster tests
* determine if certain tests can be dropped for certain types of changes
* can the asic be generated in a way that makes it simpler to test?

son of parnas
Wednesday, September 01, 2004

The ASIC company I worked for at one time used a variety of different techniques, including leasing a really expensive hardware simulator (some kinda FPGA-based monster, I gather). If your chip is strictly digital components, you might be able to use something similar. Your EDA tools vendor can probably put you in contact with a simulator supplier.

On the not-spending-a-million-dollars-a-month front, your existing simulator presumably has hooks to call out to native code routines. You could make a test harness that replaces the parts of the chip that aren't being changed/tested with fast libraries that do the minimum to test the changed functional block. You don't need to simulate the whole thing if parts of it aren't changing.

Other than that, it's just good software engineering principles.
1. Don't try to be too clever at first. Make each functional block work correctly, then optimize.
2. Don't change stuff you don't understand.
3. Code review is your friend. At least two people should look at any changes before they're committed to source control.
4. Avoid unnecessary coupling between blocks. Of course, you're designing hardware, so this might always be very practical advice.

One thing I've heard of, but haven't seen used, is translating the HDL to C code and compiling/running it on the development workstations, or vice versa. Google "HDL to C translation" for some starters.

Mark Bessey
Wednesday, September 01, 2004

Milestones every 10 days or so ?

Wednesday, September 01, 2004

Hmm sounds about right. System level testing on ASIC designs has always taken all day. When I last did any ASIC design the toggle coverage and stuck at fault modelling took all day to run even on the monster simulator that Toshiba had attached to their mainframe.

Peter Ibbotson
Thursday, September 02, 2004

*  Recent Topics

*  Fog Creek Home