Fog Creek Software
Discussion Board

Distribute Tests Across Dev Tean PCs; Good or Bad?

To save costs and keep the PC box-count low, the boss wants to make more effective use of the wasted CPU cycles inside the company by distributing the automated nightly builds and tests across the Dev teams desktop workstations.

Some of the automated tests are purely software based (simulators, etc), but some of the nightly tests will require additional hardware to be put in the Dev PCs to facilitate testing our custom hardware as well; mostly PCI hosted serial ports and relay interface boards.

The boss reminds us that our dev-PCs are His companies property, and thinks software developers are too emotionally attached to the corporate PCs they work on!

Is this a quality-control disaster waiting to happen, or what?

I’m Interested in your opinions on the subject, comments any one?

Control Freak
Monday, September 8, 2003

Developer PC's are usually not "standard installation" PC's. And in my experience usually  filled with all kinds of tools, utilities, beta's, testprojects, etc.
Ok for night builds? Yeah maybe.
Ok for testing? Definitely NOT

Testing, automated or not, should be done on machines with identical configurations as the intended production environment. Dev-PC's do not qualify for that.

Geert-Jan Thomas
Monday, September 8, 2003

I don't know that it's a QA disaster, but your developers may be the type who change their machine configs depending on what they are doing. In that case, you may have all sorts of 'build failures' come and go that are related to the wierd configuration on a particular developer's machine.

Isn't this going to interfere with regular development activities? I've been known to leave tests running overnight on a regular basis - don't your folks need to do that?

Your boss is correct, those are his machines, and should be used to the betterment of the organization. On the other hand, your boss needs to understand that everything has a price, and if he damages the productivity of the dev team, then he may be worse off than if he'd never thought of doing this in the first place.

Just my two cents...

Michael Kohne
Monday, September 8, 2003

It is certainly possible, but has your boss considered:

What if Developer needs to work late while a test is being run?

If these are windows boxes what if person trying to run a test and a person trying to develop need the same box?

What if test destroys a developers PC?

What if developer destroys (does someting to accidently ruin the OS installation) a box a test was on?

Is the time taken to administrate the changes (isolate DLLs, create new accounts, downtime mounting extra boards etc ...) really that much less valuable than new pc's?

if tests could be run on different OS with dual boots, who gets priority?

Does your bos realize decent pc's can be bough for 400 dollars each?

Daniel Shchyokin
Monday, September 8, 2003

"Testing, automated or not, should be done on machines with identical configurations as the intended production environment".

Really?  Why?

What if the intended production environment is a field of a million customers, all of which have wildly different OS configurations?  With a QA monoculture, you might be missing lots and lots of real world bugs.

Seems to me running tests on development machines would be a good alpha deployment.

There are some things to worry about, but none of them are real showstoppers for this sort of endeavor.

The boss seems like a bit of a tightwad though.  How much is he paying his devs?  Let's say $50/hr.  If by this initiative each dev loses 8 hours of work maintaining these test machines, he's really not saving any money.

Monday, September 8, 2003

Reminds me of the days back in grad school where folks would run their multi-day simulations across the bank of Solaris machines.

If the person was an undergrad, you niced them out of existance. If the person was a prof, you niced them at your own risk.

If the person was a grad student, you went by how often they had made new coffee. If they never had, then good luck getting cycles on my machine!!!

Monday, September 8, 2003

1) does every dev have two machines? one is the tweaked-out dev machine. the other is a fairly plain 'email' machine, which doubles as the 'test' machine.
2) make it optional. this works especially well for stress type testing if you can partition inputs across machines with variable configurations.
3) multi-boot. if the dev never needs remote access (files on server, remote desktop), have a 'test' and 'dev' boot.

Monday, September 8, 2003

1) Distributing the automated nightly builds across the Dev teams desktop workstations.

I've heard nothing but great things about Xoreax's IncrediBuild (distributed build software) from shops that have tried distributing their automated builds.  Apparently, it cut's down build times immensely without being very disruptive at all.  But I've never seen it in action myself ...

2) Distributing tests across the Dev teams desktop workstations.

Hmmm ... now this sounds messy and potentially disruptive to the dev team. 

Immature programmer
Monday, September 8, 2003

They sure as hell better run off of clean code from the repository, and not the developers working copies!  Plus, I agree that devs are likely to tweak DSNs and registry entries and installed files and the like, which could completely screw up a test.

I have a better idea, run the tests on the managers' and support staffs' machines.  They are far less likely to have been tortured by developers.  (the machines, that is)

Keith Wright
Monday, September 8, 2003

I think it's a good idea to run a distributed build (meaning compile). This speeds up the development process. Personally, I would rather have a faster compile than a faster computer. Tests are different since developer computers could have a ton of software and tools and who knows on it.

Tom Vu
Monday, September 8, 2003

One of the clear problems that appear is this:

The application you compile usually needs a runtime: some .DLL files, a DSN, a certain version of Internet Explorer, etc, correctly installed.

The devs will have these things already installed, so the errors about not having them will not show up.

So, in order to have a good release, you MUST make sure to test on virgin machines, with the OS and nothing else installed - so you see exactly which components of the app are missing from your installation.

Also, it is good to test on several virgin OS machines, with several configurations, etc, etc.

Tuesday, September 9, 2003

*  Recent Topics

*  Fog Creek Home