when to do perf/scalability testing?

I'm having a bit of a debate with one of our engineering leads, would be interested to hear others' views & experiences:

Like many companies, we declare each new software release to have reached "GA", General Availability, when it is substantially free of major bugs (or when customer pressure is sufficiently high  ;-)    But we seem to often *start* scalability and performance testing at that point, whereas I had believed/understood that perf testing should be a prerequisite to GA.  Is it normal to declare something GA and then commence with perf & scalability testing?

Baldwin Mills
Friday, April 23, 2004

That's certainly one way to do it,  though if there's something *seriously* wrong then you're in trouble.  A former employer of mine did this,  basically they started the roll out once all major defects were fixed, then started stress testing.

The app started falling apart under 1/5th of the intended production load.  The roll out was halted,  heads rolled etc. etc.

I've always felt that you should at least give your app a basic perf test once the end to end process is working for a few of the functions.    Then if you wait till giving it a proper test, you've got *some* certainty.

anon for now
Saturday, April 24, 2004

Load testing before release is a very good idea. However, if your app works fine under the intended load, stop. Don't attempt to optimize where it isn't required.

Brad Wilson (dotnetguy.techieswithcats.com)
Saturday, April 24, 2004

Do you have requirements for performance and scalibility?

If not there is nothing to test.  Though if there were no
requirements exactly how did you create an architecture?

The GA test suite must include tests to ensure all
requirements are met. These tests should
be run as part of the normal test cycle.

So no,  it's not normal or right.

son of parnas
Saturday, April 24, 2004

If your user base is large, then it is a better idea to do the Scalability Testing as part of Integration Testing or Acceptance Testing. Because responsiveness of the application can dramatically change from the Testing environement to Production environment.
we have a script based Stress testing for every application as part of the Integration Testing.
Stress testing the entire functionality may be time consuming, but the functionality commonly used by the users should be stress tested.

Also, profiling the code since the beginning of the development, helps a lot in case of performance issues later. profiling code can be macro based, so that it won't effect the performance at all.

mrc
Saturday, April 24, 2004

> profiling code can be macro based

Sorry to stray, but what does that mean? I assume that maybe you put placeholders in your code that are turned on and off by doing some sort of S/R.  I was talking ot a cohort about the profiling I put in the code while I'm developing. I take it out before I release it. I don't like the idea of wrapping it in if statements but he thought that would be okay.  I could write a perl script to open and s/r places where I put certain comments to set/unset markers for my profiler. Am I on track here or just totally off base?

me
Sunday, April 25, 2004

I'd say you have to test only when you know it is otherwise ready. Otherwise what are you going to profile? Code that doesn't exist yet? Potentially incorrect code?

Monday, April 26, 2004

The earlier you start perf testing, you make life easier on the devs who need to fix any perf problems.  If you do a week to week perf run, measuring deltas, then if perf drops one week, dev knows that it's probably due to something they did the week before.  It could be something else also, but you start with only one week's worth of changes to consider, versus having to look at the whole application at once.

a
Monday, April 26, 2004

You should be testing the implementation feasability of the performance requirements.  To pick a random example, I've heard of this chat application where they didn't realize until code complete that their implementation + target user base requirements would produce 3000 hits a second on their web farm.  Catching that up front would save a bit of money, no?

Anonymous
Monday, April 26, 2004

> If you do a week to week perf run, measuring deltas, then if perf drops one week, dev knows that it's probably due to something they did the week before

That assumes that you want to try and code your performance into the software from the start. My opinion is that the only thing that will happen fast in this case is losing your hair. Wait until you've finished, test, profile, tune, repeat until done.

Tuesday, April 27, 2004

  Recent Topics

  Fog Creek Home