Fog Creek Software
Discussion Board




how relevant is optimization?

I came across this quote recently while reading an article on codeproject:

"And in almost all cases, for almost all parts of almost all programs, classic optimization no longer matters!"

The subject of the article was resolving errors that pop up when compiling a release version of a project versus a debug version. The point of the quote above was that sometimes it's more efficient to simply turn off a particular optimization that is causing an error rather than spend time debugging and fixing the problem in your code.

I was thinking about optimizations in a more general context, however. How important is "classic optimization" these days? How often does the time spent on say, hand optimizing a section of code to improve performance slightly, pay off in terms of ROI of that time? Do many developers feel that "classic optimization" is a waste of time with fast processesors and such?

Ben
Sunday, July 27, 2003

Pick a design, build it. When users complain about something being slow, fix it.

Philo

Philo
Sunday, July 27, 2003

Philo put it in a great way, IMO. Write it first, and if proves too slow, optimize it.
  Unless you're an Assembler guru, you will have little luck out-performing a modern-day compiler -- it's simply too 'good'. Of course, this is a matter of opinion.
Nevertheless, you'll find that the techniques used in the early [and late] 90's don't apply any more-- the CPUs of today are so fast it's hardly worth the effort

Mickey Petersen
Sunday, July 27, 2003

Philo, i'm gonna have to disagree.  Not all things scale in a linear fashion.  I've had times where I wrote something that was blazingly fast, but oh...waith, when there are 500k users a day hitting the site....it starts to crawl.  Or when there are over 4 million rows in the database. Or you have more then 100 simultaneous transactions.  Sometimes slowdowns come out of the blue.  I think its important to unit test the holy shit out of things that may have possible slowdowns.  Sure, don't do it if it doesn't matter, but you do NOT want to have to redesign something when you have thousands of customers every day unable to hit your site or use your app.

Vince
Sunday, July 27, 2003

In most cases today optimisation isn't the case of optimising for CPU anyway, it's optimising for memory that fits into a particular level of cache.

Memory is that much slower than CPUs that cache misses kill your program's performance much more readily than a few extra instructions.

As far as not debugging your program and just turning off the optimisation, I don't recommend it except as a last resort. Your program obviously has some kind of problem, so do you really want to send it out into the wild and just hope turning off the optimisation has fixed the problem completely? I'd rather not.

And the horse you rode in on
Sunday, July 27, 2003

"I've had times where I wrote something that was blazingly fast, but oh...waith, when there are 500k users a day hitting the site....it starts to crawl.  Or when there are over 4 million rows in the database. Or you have more then 100 simultaneous transactions."

Do these things happen overnight? I've got a half million rows in my busiest table (since Feb), and I profile it about once a month. As the growth accelerates, I'll watch it more carefully.

BTW, there was an assumption that we're talking about an experienced developer - someone whose design is likely to be a "top ten percent" design anyway. For 99.99% of the applications, top ten percent is good enough. For 99% of the rest, the need for greater scalability will creep up on you, it won't just appear Monday afternoon - so you'll have time to find the flaws and deal with them. For the rest - well, I hope nobody was going to take my advice to just "throw something together" when redesigning the NYSE data farm. ;-)

So, given that (by my guess) 99.9999% of applications will be well served by a developer's best design, is it worth making the project take three times as long to shave another ten msec off a query?

Philo

Philo
Sunday, July 27, 2003

I don't know who said this, so pardon my paraphase: "If you spend all your time optimizing 10% of your code, the remaining 90% will become the bottleneck". This seems to apply when people do localized optimizations in an area they aren't sure needs it.

Mickey Petersen
Sunday, July 27, 2003

Don't worry about keeping code 'fast' or even picking a perfect design. Just keep things seperated. Make sure everything with an 'N' in it (N concurrent connections, N records, N gigabytes, etc) is factored out into its well defined little corner, so that you can rewrite it if necessary.

Most things won't need to be rewritten. Just keep an eye on things and make sure it'll be easy to rewrite if it needs it.

Premature optimization is the root of all evil. - Don Knuth.

Mike Swieton
Sunday, July 27, 2003

In client server apps, as long as it isn't _really_ slow I don't bother unless the user complains. My hourly rate is roughly the same as a 512mb stick of RAM - from the clients perspective its generally more cost effective to throw hardware at the problem.

Rhys Keepence
Sunday, July 27, 2003

I think you need to keep the 20,000 feet view of optimization when designing and initially coding a project. 

All I mean, at this stage, one of the factors to consider when choosing between different approaches to a problem, is avoiding those approaches which are "stupidly slow", for only things that'll show up [so don't bother about even stupid slowness in screen painting in a I/O bound app, but do bother about avoiding idiotic methods of querying a database or whatever].

Early on, though you primarily want to focus on modularity and correctness. These are plenty demanding enough.

Once you have a correct and modular and *complete* design, then you can look at more specific optimizations. I spent 1/3 of the time (total = 2 months) on my last project optimizing - and from this and previous experience, I can say you are 100% wasting your time if you don't measure. All those micro-optimizations (little C tricks) are a waste of time if (a) you don't measure, (b) aren't attacking the right problem.  Because of (b), I'd say it is always better to write readable code initially rather than tricky, supposedly-fast, code.

Be also aware that when you do get around to these micro-optimizations, you might end up needing to break or bend modularity.  Which is another reason to do it last.

Optimization can and is a selling point for many apps.  I know from direct personal experience.

And finally let's not forget, optimization for CPU performance is only 1 kind of optimization. Some apps need to be optimized for network bandwidth, database load, or other factors - and again these can be selling points.

S. Tanna
Sunday, July 27, 2003

"In most cases today optimisation isn't the case of optimising for CPU anyway, it's optimising for memory that fits into a particular level of cache."

I've seen this same theme repeated on here several times, and I believe that it all relates back to an MSDN article talking about the overhead of managed code, with some rather rudimentary benchmarks, where it then segued into cache sizes and memory performance. I believe that many seem to be taking this article grossly out of context, just as I believe that the conclusions of the article were grossly simplistic (and is trying to fool people into believing that you needn't worry about what's between you and the hardware anymore -- just that you keep your array sizes small enough).

Most software works on small data sets that exist within a cache, and where it doesn't it often works on sets large enough that the cache has loaded with the required data through prediction by the time it gets to it. The only time some major memory speed roadblock is hit is generally when someone is making an artificial benchmark that jumps around in multi-megabyte datasets just to prove that artificial scenarios hit memory roadblocks. The real-world usefulness of such examples are pretty trivial.  Instead with prediction and faster bus speeds, memory for real-world uses has kept up, which is why application benchmarks show a smooth improvement up until the very latest CPUs.

Dennis Forbes
Sunday, July 27, 2003

No they don't ... look at any benchmark site, and you'll see that a new CPU using the same design does not scale anywhere near MHz for MHz. Many benchmakrs it's a matter of a 10% increase in MHz is getting you only a couple of percent increase in performance.

Yet when the bus speeds are upped (memory bandwidth), you see big jumps in performance.

The thing is, anything that's not working on large data sets is unlikely to be CPU bound these days anyway. For the stuff that CPU speed matters for (mostly games, rendering, audio/video work, etc), you're using huge chunks of data.

Databases on the other hand are primarily bound by how fast your disks are, unless you're lucky enough to be able to fit the whole thing into memory. Fortunately, you're very unlikely to be writing a database unless you work for Microsoft, Oracle or IBM. ;)

And the horse you rode in on
Sunday, July 27, 2003

Gee, I suppose if you mean by classic optimization you mean memory requirements, and also code speed, yes..that does hold true. Both lack of processing and lack of memory used to be the two most common problems.

Now, the above hardly matters.

The disk to memory speed, and network speeds are the precious commodity to day. Loading programs from disk is where most users experience waits right now. The same thing goes for waiting for network requests.

So, in effect to day the problem is bandwidth (disk to memory, or network to memory).

Processing speed and memory we have plenty of.

How fast can you get stuff from the disk, or how fast can you get stuff from you network connection is where much optimization occurs today. When either of these waits get to long, then users start to complain (you then modify things to load less etc).

I don’t know of any user compiling about not having enough processing since about 500 or 600 MHZ (sans the gaming issue). 

Albert D. Kallal
Edmonton, Alberta Canada
kallal@msn.com
http://www.attcanada.net/~kallal.msn

Albert D. Kallal
Monday, July 28, 2003

It's common to find programs that employ O(n^2) or O(n^3) algorithms where an O(log n) will do -- just because it was easier to write at the moment and fast enough while debugging (consider that, assuming the constant is in the range of 1000 cycles, an O(n^2) algorithm takes around a second for n=1000- which most programmers will accept). At a later time, the required 'n' becomes 5000 and the CPU only goes from 1GHz to 3Ghz in the meantime, the 8 secs are no longer acceptable - but the programmer(s) have moved on and will not fix it because it's been a year.

Also, I've been called more than once to help optimize a project that took the "first get it working, and optimize bottlenecks later" approach - and in many of them, once the main bottleneck was removed (and this wasn't sufficient), there was no bottleneck to solve - everything was more-or-less equally inefficient.

If you ignore performance issues in a few places, but think about the overall thing, you'll probably be able to speed it up when the time comes. If you ignore performance everywhere, there's a good chance there won't be anything small enough to fix that is worth its while.

There are a few things that you can't put into a project in retrospect: Robustness, Structure, Speed, Security and possibly others.

Those of you who say "will fix the speed later" probably think you can fix robustness and security issues later as well. Collective experience in the software industry is overwhelming to the contrary about robustness and security; And despite the "common wisdom" about optimizing later, my experience leads me to believe that, if you don't design it to be optimizable in the first place, it won't be optimizable if you need to optimize it, just as you won't be able to secure it later if you didn't design it to be secure in the first place.

Ori Berger
Monday, July 28, 2003

Ok, you guys have swayed my opinion.  I'll now say, with good object oriented design and good architecture, optimizing later on becomes trivial.  I would say good architecture is more important; when you have a bad design, optimizing later on becomes almost impossible. 

Also, philo, your right, you should take into account the project your working on, and do a cost/benifit analysis.  For "philo's ecommerce baked goods site", I doubt its worth it to do unit testing and make sure it can handle a couple million transactions.  If your designing something you *know* can't slow down, you should probably spend the extra time/money to optimize, or at least plan for optimizing at some point.

Vince
Monday, July 28, 2003

When you say "good object orientated design", I assume that you realise that there is a mis-match between relational data and objects. This means that sometimes you have to sacrifice some of the "goodness" of your oo design in order to optimise your database access.

I've seen a database driven app increase it's speed by a factor of 100 by switching to a less "ideal" oo design. This did however require a redesign and rewrite. Database access is expensive - it pays to always design with that issue in mind.

RB
Monday, July 28, 2003

"The thing is, anything that's not working on large data sets is unlikely to be CPU bound these days anyway. For the stuff that CPU speed matters for (mostly games, rendering, audio/video work, etc), you're using huge chunks of data."

I specifically included the disclaimer that those applications that ARE working on large enough datasets often actually do something with each batch of data as it pulls it, giving ample type for prediction to grab the next set of data as it gets around to it. Without any doubt bus speed and cache size matter, however imaginations that the CPU is irrelevant are just that: Imaginations. I can compress a massive DV-AVI movie to a WM9 video dramatically faster on a Athlon 3000+ than I can on a 1800+.

Dennis Forbes
Monday, July 28, 2003

> The subject of the article was resolving errors that pop up when compiling a release version of a project versus a debug version.

We shipped with compiler optimizations disabled, for two reasons: having only one build means that it's better tested (you don't want to situation where the only people who run optimizations enabled are QA and customers but not developers); it's easier to debug crash dumps from the field (for example, optimizations enabled can mean that you don't get a proper call stack).

We broke this rule for only very few modules (for example, the image processing library had optimizations enabled).

In our experience most all bottlenecks were algorithmic: for example, thread contention for critical resources; getting too much or too often from SQL; sending too much over the network; using inefficient components from other vendors; using an STL container that made too many calls to "new".

My rule of thumb was that anything you do in your own code (except iterating data) is negligeably fast compared to the subroutine calls that you make into the O/S, the RTL heap, and devices.

Christopher Wells
Monday, July 28, 2003

Ori Berger wrote:

"...in the meantime, the 8 secs are no longer acceptable - but the programmer(s) have moved on and will not fix it because it's been a year."

Wow, what a completely inappropriate reaction from a developer.  Who cares how long it's been?  Code doesn't rust.

"There are a few things that you can't put into a project in retrospect: Robustness, Structure, Speed, Security and possibly others."

Completely disagreed.  Assuming that your program is well-written, and that you've removed all duplication, you can put security and speed in later.  Isn't the whole point of optimization to increase speed?  Are you saying optimization is impossible?

The Pedant, Brent P. Newhall
Monday, July 28, 2003

>> Processing speed and memory we have plenty of. <<
Obviously you've never heard of PDAs or smart phones. 

SomeBody
Monday, July 28, 2003

Ori, I think you are misunderstanding what people are saying.  Yes you avoid putting in stuplidly inefficient code, if there is no reason for it.  But there is no reason to heavily optimize before you can say what code is the bottleneck.  You're just wasting your time if you try to do a lot of optimization as you go.  That's not to say you shouldn't pick more efficient algorithms if the options are otherwise equal, but you shouldn't waste a lot of time thinking "this code is not as efficient as possible, I should optimize it now" because it is very likely that the optimization will be unnoticable.

Algorithmic/structural optimizations are definitely far far more important than assembly optimization.

One last note, I think that the quote may partly be referring to specific examples of "classical" optimization.  For example, "loop unrolling" is a typical classical optimization, that can actually hurt performance in modern chips.  Modern chips have good branch prediction, making loop unrolling less of a benefit.  At the same time, expanding your code risks cache misses, which cause huge performance hits.  It's these types of "classical" optimizations that are no longer relevant, because the architecture has changed so much.

Mike McNertney
Monday, July 28, 2003

Brent - I'm surprised you've never received such a response. Many times, (especially in, but not limited to, the present state of the economy), a project that goes into maintenance mode will only get crucial fixes or billable fixes. Speed improvement doesn't usually fall into one of these categories. One might argue that in this case, the speed improvement is not worth the extra effort in the first place; I like to think it does, from the client side, as it does improve (significantly) the product's longevity.

And while code doesn't "rust", the cost of modifying it increases with the time since it was last actively maintained. For a variety of reasons, such as, the original programmers are no longer available, or forgot most of the issues, or the environment changed. Just a few days ago, I had to use a Java API that for some reason I don't understand is only compatible with JDK 1.2; Not 1.3, not 1.4 and not 1.1. This code (and code that uses it) are harder to support now, because the development tools target 1.3 and 1.4. I can't say its rotten, but it's definitely stale. But I digress...

Brent, Mike, I do not mean that one should optimize each and every function. But one should consider what should be optimizable, and make way for it in advance - or there's a good chance it won't be possible when needed.

Some design decisions make it impossible to optimize later. For example, if you mandate SOAP as the function call protocol, and independently require a call to be iterated for a million objects, then there's a good chance you won't be able to optimize this later (There's an emerging market segment of XML parsing accelerators that have dedicated processors because of such stupid design decisions).

Consider, e.g., an OLAP browser (such as hungry dog software's IntelliBrowser), that feeds of data it reads through SOAP through XML from various locations. OO purists will probably admire this design as it abstracts and decouples everything, and allows one to distribute objects cleanly, letting the SOAP/XML layer do everything. And you know what? It'll work for a small number of objects (in the thousands range). But you can't speed it up without significant architectural change. I'm not saying that the initial implementation must be as fast as possible; I'm saying that the implementation must take future optimizations into account or they won't be possible as local changes.

Brent - do you know of a product in which security and robustness were added after it was feature complete? Most products I know are examples that this CAN'T be done. Sendmail and Outlook have been audited for security god-knows-how-many times, and yet for years new vulnerabilities are discovered. Qmail was designed to be secure and it is.

I've got an interesting example about "how relevant is optimization" in the classic sense - I'll post it under a new topic in a few minutes.

Ori Berger
Monday, July 28, 2003

*  Recent Topics

*  Fog Creek Home