The Old Joel on Software Forum: Part 4 (of 5)

End to End Testing?

In our organization, a person is pushing for End to End testing of software changes.

Does anybody have any experiences (Stories) to tell about this?

This does not seem reasonable for our large system given the current manpower.

We can test the change or enchancement that was requested, but re-testing every item in the system for every change seems unreasonable.

M
Friday, May 28, 2004

This is a pretty contentious topic...

Of course, you'd always liek to be able to provide complete end to end testing whenever a change is made... but realistically, it's difficult unless you have (near) fully automated test procedures.

While Test Driven Development, eXtreme Programming, and numerous other methodologies have this as a goal, it's difficult in practice.

KC
Friday, May 28, 2004

Depending on the project you can treat it like
triage. Perhaps on some products it is possible.
If your setup is long and varied and/or there
an immense amount of tests then it can't be done.
Cisco may run months of tests for a release.
You can't do that for each change.

1. Run unit tests. Run for every change.
2. Have a sanity test that runs large scale functionality. Run every day.
3. Have system tests that tests every day. Run these as often as possible, where it makes sense in the release cycle.
4. If you have vendor acceptance test plans then run those as well.

son of parnas
Friday, May 28, 2004

We do partial end-to-end testing (is that really end to end?). Partial meaning that its difficult to get 100% coverage. The server side can be made to 100% coverage, but the client would be prohibitive. Its a continual work in progress. I have very little to do with it, but I'm thoroughly impressed. While its primary purpose is to catch bugs, it works beautifully as a developement too to measure and benchmark performance. Something like it is required for verifying scalability - even if coverage is not complete (all scenarios covered).

hoser
Friday, May 28, 2004

For your automated tests, this is a very reasonable goal. For any tests that are not automated, you should look into the possibility of automating them (I don't believe in automating UI testing, but do believe in automating the layer just below it).

If you add in a continuous build process (see previous mentions of tinderbox, anthill, and cruisecontrol), then this can be a painless and worthwhile thing to do. Unit tests are nice in order to verify individual components act as desired, but they do not validate that the component integrates correctly with all the other components. For this you need end to end testing.

Even seemingly minor changes can have major effects. The longer you wait before you detect those effects, the more code you have to dig through to determine what exactly caused the problem. If you can check in small discrete changes, and have the build server verify as much as possible what code is causing problems, you will be much better equipped to know where the problem lies.

Oren Miller
Friday, May 28, 2004