Fog Creek Software
Discussion Board

Quality Improvement via VBScripts

I've been tasked with designing and building a new product for my company, which just so happens to be an NT Service that exposes several COM objects for managing the service.  I am working alone and under the gun, ( I don't have much time to do it) so I'm trying to adopt any practices which can help prevent me from making brain-damaged mistakes.  One idea I arrived at is as follows:

After implementing a chunk of functionality for a COM object, write a VBScript that excercises the method(s) on the object as well as possible.  Make sure to test for expected successes as well as expected failures.

Next, place these VBScripts in SourceSafe (or whatever you use) along with your project.  In Visual Studio, create a Custom Tool item that executes the VBScript directly from Dev Studio.  (If you're not sure how to add Tool Items,
please select Tools | Customize from the main menu in Visual Studio 6.0. )

Now, adopt, as I have, the will to "not check stuff in" until your little VB Scripts pass their tests.  This might seem like a lot of trouble, but it has already saved me some headaches.  I've already found a few cases where, for instance, I was blindly returning S_OK in some method when in fact something failed. ( The ATL COM Object wizard typically generates stub implementations of methods that simply return S_OK, and I missed a few of them.)

I'd be interested to hear thoughts on this idea, as well as other ideas for ensuring quality despite crazy schedules.

Thomas R. Dial
Tuesday, February 18, 2003

Welcome to the wonderful world of unit testing.  I try to write tests for everything I do and it really saves me a lot of time.  I don't work late nights any more for a reason, and unit testing has a lot to do with that.

Joe Blandy
Tuesday, February 18, 2003

>> Welcome to the wonderful world of unit testing

Well, we'd been unit testing with scripts for awhile, but I've been working to find ways to make it easier to do, and adding the simple Tools customization was just one I'd
come up with.  I might actually write a program that uses
the ScriptControl OCX to blast through a suite of tests
all at once and generate reports.  This suite could be
run before every check in.

I doubt it will keep me from working late nights, but at least I'll writing more solid code.  My philosophy has
always been (regarding code) that I never want to see
it again once I'm done.  This isn't a bad attitude as in:
"I won't fix bugs."  It's a good attitude as in:  "I never want to have to fix bugs, so I'm going to crank the level of quality up."

If anyone has any other cool ideas, let me know.

Thomas R. Dial
Tuesday, February 18, 2003

Using scripts to test the COM component would be a black-box approach. I think you would be better off with unit tests that can access the internals of your component, i.e. written in C++ (or whatever language you're using.)

Personally, I'd keep COM out of the core functionality of the component to keep it simple, and test it through that interface, then hook a COM interface on to it for production use. I still suggest you write VBScript tests, but only to verify that the COM interface correctly exposes the component's functionality.

You don't mention having a functional specification or a detailed task list. I'd recommend doing both of those before writing any production code. With a short schedule, it shouldn't be too hard to plan your work in detail, which will minimize the time you spend refactoring as you go. They will also help you prioritize so you can get the critical tasks done first. I wouldn't be afraid to spend about one third of the total scheduled time on planning.

Big B
Tuesday, February 18, 2003

Big B,

I really don't see why using VBScripts is a black box approach.  I always thought that the term "black box"
implied a far-less detailed understanding about what events were causing what effects.  If my script fails,  I know why because I'm targeting very small pieces of code with the script.  If I need more detail, I can simply attach a debugger to the service process, or if it is a DLL, I can set the Windows Script host as the Debug Executable.  Either way,
using VBScripts does not keep me from stepping through to the C++ code, and it is definitely faster to crank out a script than to write  a C++ program test my C++ program.  I know
that it probably keeps some programmers away, I've seen many an interns eyes glaze over when told to attach to a process and step through some code, only to go back and find them printf'ing or somthing like that.

I understand why you might say "Personally, I'd keep COM
out of the core functionality of the component to keep it simple.."  but my experience has been contrary to this.  I've found that writing in, say, C++, and then "wrapping" with COM leads to other headaches.  I can understand where this strategy might be useful if writing for multiple platforms.  For example, write a basic API in C and then provide C++ wrappers and Java wrappers through the JNI.  I mentioned headaches-- what do i mean?  By wrapping the code with COM, you introduce more opportunities for dumb bugs, especially since the wrappers are boring to write and may be done carelessly.  You might argue that templates and Macros could partially mitigate this risk, but in my experience, wrapping sorta sucks, and in this case where I'm not targeting other platforms, it adds complexity WITHOUT adding any real benefit.

Another thing about wrapping: it will cost time.  The ATL
COM AppWizard can make an NT service (which is basically
boilerplate) in about 1 minute.  Adding / Exposing COM objects is a snap.  Why deprive myself of these tools?  If I'm going to expose COM, why not put the guts in the COM code too?  Many times, the only reason wrappers exist was so that software vendors could leverage legacy code.  But then they got in the mindset and felt like everything should still be written in C or C++, and then wrapped.  At least that's my theory.

I really liked what you said about having specs / planning ,etc,.  I actually spent more time than normal on this project and because nobody else would, I wrote a detailed specification of the product myself, and got the big boss to okay it.  Next, I did a block diagram, still my favorite tool in the world, and started hashing out the interfaces in a style similar to UML.  The weirdest thing I did, as sort of a sanity check, was to write a (practically) stream-of-consciousness
type document describing a typical "use" of the software interfaces.  I uncovered some tricky things with this.
My last step was to finalize the "classes" and their properties and methods, and as I code on this, I haven't had to make one change yet.

Thanks for your input, Big B-  my degree was Economics, not Computer Science, so there are a lot of terms and stuff that I don't know.  That's why I love this web site because there are so many expert perspectives.

Thomas R. Dial
Wednesday, February 19, 2003

Here are some additional thoughts:

If you write your tests in VBScript, you can only test your component at the level of granularity that the component exposes. You cannot test internal functions independently. I would consider that integration tests without unit tests. But then again, our vocabulary might differ. In any event, you may find it easier to target the internals of the component directly rather than through the outer layer.

It shouldn't be much work at all to write tests in C++ if you're using a unit testing framework such as CPPUnit.

If I were very confident with ATL, COM and the code generating wizards, I might consider implementing the logic directly in the generated code. If the wizard at any time gets out of sync with the source, and you need to start over, it will be a lot easier if the guts of your component are not intermingled with the wizard-generated code. Other issues such as readability and versioning also come to mind in favor of separating the two.

Your argument about making mistakes because the process of mapping the COM calls to the component's interface is "braindead" doesn't work if you're creating VBScript tests to verify the interface mappings.

Debugging an external process can be useful, but in my experience not as convenient as debugging a regular executable from the IDE. I am not much of a VC++ user. Sounds like your experience may be different.

Anyway, what really matters is that you seem to be taking responsibility and care for the quality of your product, and I'm sure you'll do great whether you take my advice or not.

Big B
Wednesday, February 19, 2003

*  Recent Topics

*  Fog Creek Home