The Old Joel on Software Forum: Part 3 (of 5)

Testing Question

Is Regression testing the best method for testing APIs?

Just in Time
Monday, January 12, 2004

No.

Regression testing finds regressions: in other words, it finds cases where things that were known to work in the past are now broken.

If your existing test cases (i.e., the tests you use for regression testing) have holes in their coverage, however, you can still have tons of bugs in your APIs that you never catch.

Good API testing will start with writing a thorough set of tests that are likely to find bugs in the APIs - whether or not you run them as "regression" tests is besides the point.

Also, once you have a good set of regression tests, you can still benefit from continually coming up with creative new tests to do mean things to the code that you never thought of when writing your previous tests.

Mike Treit
Monday, January 12, 2004

Let us know if you pass the take-home technical screen and call you for a real interview.

Alyosha`
Monday, January 12, 2004

I shouldn't be so sarcastic. Half the job is knowing where to find out the information to stuff you don't know. And JOS is as good as place as any.

But then again, STFW.

Alyosha`
Monday, January 12, 2004

STFW? means?

Aussie Chick
Monday, January 12, 2004

STFW = Search The F*cking Web

G'day, Aussie Chick!

WooWoo!
Monday, January 12, 2004

As opposed to STFU, which I though he wrote.

(Shut the F* Up)

Don't let Steve Irvin near your baby!

pdq
Monday, January 12, 2004

Gee, now I feel kind of embarrased for trying to give a reasoned, well-thought-out answer.

Guess I'm a sucker for testing questions.

Mike Treit
Monday, January 12, 2004

Hey Mike,

Don't be embarrassed for trying to help someone. You the shizzle, baby!

WooWoo!
Monday, January 12, 2004

"But then again, STFW. "

And thus JOS begins the slow, downward spiral to another SlashDot.org web site where rude, smart-ass posters grief people who ask less than perfectly intelligent questions.

Go to Slashdot you asshole.
Tuesday, January 13, 2004

Mike - thanks for your answer. I learned something from it, even if it was a homework question.

nathan
Tuesday, January 13, 2004

I don't think the original question makes sense, because there's not enough information to give a sensible answer.

What sort of API is it? What release is it on? What parts are under change control? How many dependencies does the API have? What kind of regression test were you thinking of - scripted or manually driven from a spec? If the latter, what skills does the tester require?

Better Than Being Unemployed...
Tuesday, January 13, 2004

Actually there is a sensible reasoned answer to all this. As posted above, regression testing tests exactly what it says - conformity with previous versions. That means it may be an important part of your testing or not, depending on how important backward compatibility is for you.

For APIs backward compatibility is usually very important - programmers get very upset if behaviour changes from what they expect, to the point where some APIs reproduce undocumented features or bugs that existed in previous versions. If you are responsible for maintaining an API then regression testing should be very high up your list of testing priorities.

David Clayworth
Tuesday, January 13, 2004