Testing Software - What About Method?

Hi:

I have responsible for a pice of software that includes a bunch of methods (in a number of classes) that external applications can use by making API calls.

We have a few happy coders that make ongoing bug changes.

What are the best practices for testing the methods that are exposed as APIs?

These, theoretically, are not supposed to be change as there are many external dependencies - but in practice they have been occations where changes were made and we only received the complains after the code was moved to production.

At the moment I use CVS diff to see if any changes are made - this is very tedious.

Is there a better best practices methodology to use in the QA process?

Hamlet
Tuesday, January 6, 2004

Not sure about any fancy "methodologies" for testing
API's, but having done lots of API testing over the years,
my own best way is to have a bunch of tests which exercise
each API entry point completely, and which do so hopefully
in isolation, to make sure it agrees with the spec - ideally in
both normal functioning cases and in error/boundary cases.

Each test should be short, should be a separate
executable, and should only test one thing and exit if at
all possible with your API.  Also, when testing an API, it is
a very good thing if a memory leak detector and corruption
tracker like Purify is used if these are in C/C++ - Purify can
be integrated into a testsuite relatively easily.

I usually like to have a small set of "sanity" tests that
developers can run before submitting to the source
code control system, and a large set of exhaustive tests
which are run nightly.  If it isn't enormously large, the
exhaustive suite can be run directly.

foobarista
Tuesday, January 6, 2004

In addition to what the other two said, try to develop some applications with the api. spec them as if these are your apps, another words pretend you are a customer, maybe even ask product managers for some sample customer specs

The Artist Formerly Known as Prince
Tuesday, January 6, 2004

Looks like you need some sort of change control and authorisation process as well (if I've understood you correctly). No point in creating and running test plans if the behaviour of the API is going to change whenever the developers feel like it.

PJ
Wednesday, January 7, 2004

Authorization process?  Yuck.  Processes don't produce quality, individuals do.  Processes can do only one thing: restrain idiocy.  That's a lousy thing to need.  Search deeper.

If you have programmers making careless changes to public APIs, your problem is that you need better programmers.  The minute you band-aid this problem, another one will follow right on its heals.  You probably pay too little to get good ones, or you don't know how to hire good ones.  You're in a pickle alright, but processes won't solve that.

Hire one damn good, focussed programmer and put him or her in charge, including the power to fire and replace.  (Hint: if you didn't know it, you'll have to pay at least what you pay any manager/directer/etc.)  Now, how to recognize "damn good" when you don't already have it is your next pickle.

veal
Wednesday, January 7, 2004

The only process that needs to exist (other than the
testing procedures and running them frequently) is
a clear and detailed spec on the API and, equally
important, knowledge among the programmers that
something is part of a public API or not.  The public
knowledge requirement means that the public API should be
obvious and easy to determine (ie, it's in pub_api.cpp or
whatever) and that plenty of comments and naming
conventions exist which indicate that method "foo" is
"golden" while method "bar" is not.

If programmers are still cavalierly breaking things,
some managerial processes may be needed.

Hopefully, not everything is part of the public API - a
well-designed API should be a small percentage of the
number of entry points in the system.

foobarista
Wednesday, January 7, 2004

  Recent Topics

  Fog Creek Home