Fog Creek Software
Discussion Board




Monitoring an install? Or "what did that change"?

Hi!!

As part of my "software testing" job I have to make sure the installer for the software we develop does its work properly...

We use InstallShield for the installer program, and the setup prog. has to update quite a few things (install drivers and class filter for devices, change a bunch of registry entries, fiddle with low level -MBR- settings, etc)...

What I want is to be able to know what has the setup program changed, to be sure that all the needed steps are taken and that we don't break anything we shouldn't. I know  that there has to be some software that takes "snapshots" of the system and tells you what the changes were... Or, ideally, some way of "playing" an installer and "recording" the changes it makes...

For the moment, I am trying to manage "manually". I do a listing of all the files on the HD before and after installation, and I export the Registry pre- and post- install. And then, I diff the files. But there is a _lot_ of noise in that method (plenty of files with different access times and such), and specially on the Registry, I can't never be sure if the changes are related to the install or to some other thing,

Any tip / hint on how could I do this easier (ideally, a way to automate the process would be best)?

Thanks a lot

Javier Jarava
Tuesday, July 08, 2003

I'm sure that PowerQuest used to make something that did this.  The closest thing that I could find now is their 'SecondChance' product which I couldn't even find a mention of on their site.  That monitors file and registry changes and allows you to create checkpoints but I don't know if it allows you to diff them or not.

R1ch
Tuesday, July 08, 2003

Go here and see a list of tools that may help you: http://www.sysinternals.com/ntw2k/utilities.shtml

smurf975
Tuesday, July 08, 2003

PC Magazine had a free utility that did this. It was called "InControl" and I think the latest version is 5. You should be able to download it from their website.

John Topley (www.johntopley.com)
Tuesday, July 08, 2003

Norton CleanSweep (which is part of Norton SystemWorks) does something like this -- creates a log of what changed.
http://www.symantec.com/sabu/sysworks/

Duncan Smart
Tuesday, July 08, 2003

I use Install Watch Pro from Epsilon Squared http://www.epsilonsquared.com/ does the job and its free.

Ken McKinney
Tuesday, July 08, 2003

InCtrl5 from PC Magazine. It was mentioned above. It logs everything that happens to your computer in a time period. It's extremely thorough. While it won't let you roll back becuase it doesn't make a copy of the files that change, it does tell you what files changed, what registry changes there were, etc. HIGHLY recommended.

www.marktaw.com
Tuesday, July 08, 2003

.. oh yeah, it's not free anymore, you have to have a subscription to either the magazine or to just the PC Utilities section of their site ...

www.marktaw.com
Tuesday, July 08, 2003

You might also want to think a little harder about your approach to setup testing and make sure you are making the right tradeoffs.

While a black-box approach is very often the correct one when testing, particularly when looking at the end-to-end customer experience, I don't think it has a very good "bang for the buck" factor when used for setup testing. The number of worthwhile bugs you find by reviewing the list of changed files and registry entries is going to be very low.

A better approach might be to do a thorough white-box review of the setup logic (for modern installer formats, such as Windows Installer, this can be really straight-forward) and then focus your testing efforts on any special logic that is likely to be error-prone (such as that scary-sounding MBR modification you mentioned.) You will also get implicit verification that setup worked simply by doing your normal functional testing - if setup has a (serious) bug, it will be very likely be exposed by the broken functionality.

You may still want to review all of the changes made by the setup program at some point, just to be safe, but that can probably be something you do just once before shipping to ensure you didn't miss anything obvious with your more focused, optimized setup testing.

-Mike

Mike Treit
Wednesday, July 09, 2003

Thanks to everybody for the tips...

It's a pity that the PCMag util. requires subscription, 'cause it seems to be quite the one :)

I've been looking into Install Watch Pro from Epsilon Squared and it _is_ a great time saver over the previous way of doing things. The only dobut I have is that there are two (seemingly identical) products on the site, Install Watch Pro and Install Rite, and I don't know which one is "better"... Does anyone know wich might be the differences between them? I haven't been able to "see" any, but I'm probably not looking properly.

Thanks a lot

Javier Jarava
Wednesday, July 09, 2003

RE: my approach to "setup testing" (black box vs. white box), I know that the black box method means a _lot_ more work and ooodles of "noise" (I've had to diff two 50MB+ registry exports and try to make sense of the thousands of changes! ;) I'd love to be able to review the install logic and work on from that. But there are some problems with that approach.

First of all, the person who's responsable for the installers (who is not one of the developers, but let's not get in there) says "they are fine" while there are customers complaining, so  there is a need to see _what_ the installer does vs. "what is supposed to be doing". And any suggestion of having someone else made responsable for installer generation is stonewalled, hard.

But when customers complain of "your program ate my hard drive" (which, incidentaly, it _might_ do as we modify the MBR, encrypt the whole hard drive, and a few other niceties that might kill the PC when they fail - hey, it's a _security_ software, after all!), it's the QC dept (ie, my and my two collegues) who are blamed...

I'd love to use a white box testing methodology. But as things stand now, my job is mostly doing reverse engineering of our product to see if it behaves as I've been told it should ...

As a matter of fact, what I was hoping for when I asked for tools was that there might be some program that could "reverse-engineer" an installer and list the changes it'll make without having to install it. Being able to record of the changes it makes is a great second best ;)

Javier Jarava
Wednesday, July 09, 2003

"As a matter of fact, what I was hoping for when I asked for tools was that there might be some program that could "reverse-engineer" an installer and list the changes it'll make without having to install it"

I haven't tried any of the InstallSheild ones there are quite a few 'decompilers' around that do this sort of thing - a quick google search turned up this one for InstallSheild:-

http://www.datapool.de/Windows/Tools/Decompiler/

R1ch
Wednesday, July 09, 2003

The packager in Prism { Pack | Deploy }, available from http://www.newboundary.com/sds/sds.htm, does an excellent job with finding differences in a system -- you get a single package that contains all changes that happened, including file changes, .INI file modifications, registry changes, etc.  There is a trial version that you can use for free for 30 days, I think.  Disclaimer: I work for NBT, so this is not an unbiased post.  It's a darn good tool though!  :)

David Pickett
Thursday, July 10, 2003

I think you can get a month of downloads for around $3 from PC Mag.

www.marktaw.com
Saturday, July 12, 2003

*  Recent Topics

*  Fog Creek Home