Fog Creek Software
Discussion Board

Forget design. Go for evolution

This is not intended as flamebait, but I could not help reading parts of a rather interesting discussion on the Linux Kernal Mailing List about software design. Linus claims that successfull software results mostly from evolution rather than good old solid design. While it's stated in his typical way, I tend to agree to a large extent.

<QUOTE src=Linus>
And I will go further and claim that _no_ major software project that has been successful in a general marketplace (as opposed to niches) has ever gone through those nice lifecycles they tell you about in CompSci classes. Have you _ever_ heard of a project that actually started off with trying to figure out what it should do, a rigorous design phase, and a implementation phase? Dream on.

<QUOTE src=Linus>
I _am_ claiming that the people who think you "design" software are seriously simplifying the issue, and don't actually realize how they themselves work.

Jan Derk
Monday, December 3, 2001

To design is to think before you code. It makes no sense to claim that coding without thinking first produces a better, or even equally good result. On the other hand, it's not a new idea that stiff development processes don't apply to a changing world.

Good software results only from both careful design and listening to feedback, both on a functional and technical level.

Johannes Bjerregaard
Monday, December 3, 2001

I think design isn't required to "Get the things done. Somehow. Someday.".
You may stumble around, re-writing and fixing and something (*someday*) will finally get out the door.
But it certainly doesn't work as soon as people have schedules and business plans. Linus may play around with Linux kernel for say, 2 years or 5 years, it'll probably be just fine - hey, it's Linux, who cares about schedules ?
And still - I'm sure even people producing a high-quality products without any formal requirements/design stage pass them, it may be just in their heads, similar to the Mozart keeping the whole opera there .. (that what movies say)

Does Linus start coding without thinking what is he going to do ? Without thinking what is his new kernell version is going to do ? No, but that's a design already ..

Evgeny Goldin
Monday, December 3, 2001

Ontogeny recapitulates phylogeny.  We now know that's not fully true, but the many stages of software projects are still visible within, as you take them apart.

Those who generate political forces and write code for Linux are all responding to something in the environment that pushes them to do so.  They are adapting Linux, as the environment compels them.

Of course, that does not mean we can now be stupid and not design things.  If we start doing that, then we become inefficient in adapting the software to survive.  However, does anyone think that Linux and Windows are the best-designed products ever?  They're actually badly designed, and Linus was explaining why worse-is-being-better.

Here is another interesting talk on Linux's engineering, this time by the other head honcho, Alan Cox:

Monday, December 3, 2001

I think the old quote that nobody ever builds a skyscraper without drawing the plans first applies.

I think that Linux is a special case because the design work was done long ago - the Unix operating system has been around for some time and is well understood.

Ged Byrne
Tuesday, December 4, 2001

There are though considerable resonances between successful applications and evolutionary coding.  The fuzz and furble, that Joel mentions in the recent interview, in code that has existed and changed over time.  There's a good reason for all that.  They are all applied bugfixes and they are important reservoirs of knowledge about the system.

That they are difficult to read is partly why they get rewritten out, then you get regressions between 'generations' of a product.  This all argues for a well structured bug documentation system that integrates at the source level.

Unless you can refactor the fuzz and the underlying 'clean' original design with a sense of an overall view you're better off leaving it alone.  Code reuse doesn't only apply to objects, classes and theoretical constructs.

There's a piece of software wot I wrote sitting at a client of mine that depends on a 16 bit library to convert BASIC date fields because another ancient piece of software they still use was written in old compiled BASIC and there's no good reason (until everything changes) for them to get rid of it. 

So I had the pain of writing a little C wrapper to read files and munge them for my ultra-brilliantined brand spanking new database app.  But it really wasn't that much pain and hugely satisfying.

The client will never understand the process I had to go through and there's no reason why they should.  The solution evolved to handle a new situation, I couldn't say I designed any of it.

Simon Lucy
Tuesday, December 4, 2001

I find Linus' views interesting.  I think what most people forget is there are two ways of designing things:

- Develop something for a specific purpose.
- Develop in anticipation for what's to be built on top of it.

The latter way is a statement of ignorance in the face of a complex world, and is very appropriate for an operating system.  One doesn't "execute" on a plan so much as build to create options for others to use.  Those others may design things rigorously, but the overall design is not so rigidly planned.

forgotten gentleman
Tuesday, December 4, 2001

Another thing not mentioned is prototyping.  It's a buzzword, yes, but it's been a common task is "real" engineering (the building of actual, physical things) forever.  You think that car you buy sprang directly from the CAD screens to the factory floor?  It almost certainly went through a prototype phase, where something was built, tested, discussed, and built again. 

With software, it's much easier and cheaper to iterate over working code, testing and observing as you go, than physical hardware, so why not do it?  Some basic white-board level design is useful at all levels (start, middle, end), but having working code to test theories is extremely useful. 

This model is esp. useful in systems where the distribution costs are nearly zero (like code on a website).  You can deliver an initial cut very quickly, then take baby steps towards the goal (which will always move while you're working).  At each step, you reassess, and learn about where you are, where you want to go, and how you need to get there. 

James Montebello
Thursday, December 6, 2001

Evolution works because nature has immense timescales to perfect things in and considers all products of the process completely expendable.

Can the same be said of your software project?

Directed-evolution can achieve impressive results but it requires that you have a plan to identify best-fit products and promote their survival.

In other words, you have to know what you want out of this process. Perhaps your understanding won't be 100% accurate, perhaps requirements will change but without a plan of some sort you won't be able to identify the "best fits". Better plan = faster results. You can still get there without a plan but it'll take longer, much longer.

Ian Sparks
Thursday, December 6, 2001

Evolution isn't a good analogy. Calling it "evolutionary design" implies that the programmers are running around changing the code willy-nilly without caring about the design, and then hoping that somehow the good parts of the design will stick around and the bad parts will die out. That's not what the programmers are doing.

(Well, at a lot of companies, that *is* what they're doing. But that's not what the evolutionary design guys are recommending. :)

We're not talking about hacking something ugly together and then cleaning it up. At every step of the way, you have a program that is well-designed. Then you grow that program in tiny little steps, and after each tiny step you make sure that the program is *still* well-designed. The design goals are still the same, of course - modularity, simplicity, testability, all that good stuff.

The big advantage of this style of development is feedback. Writing the code gives you feedback about the viability of your design ideas, so you learn things and end up with a better design. Developing the program incrementally means that you (and maybe even real, live users, or at least marketing guys or something :) can see the running program earlier, so you learn things and end up with a better product.

This style of development requires a lot more discipline - because, as everybody has noticed, if you do it *without* discipline you end up flailing around and creating a lousy design and taking a lot longer. You really do need to keep the design clean every step of the way, because that's what gives you the flexibility to keep changing the design. You also need to be conscientious about not breaking things while youre making all these changes.

But there are tools (like Smalltalk's refactoring browser) and practices (like test-first design and programming in pairs) that can help you maintain the discipline you need - and if you can manage it, then the results can be really spectacular.

Adam Spitz
Thursday, December 6, 2001

*  Recent Topics

*  Fog Creek Home