Fog Creek Software
Discussion Board




In Defense of Not-Invented-Here Syndrome

To outsource or not to outsource?

Is code reuse the holy grail or a recipe for disaster? Share your stories.

Joel Spolsky
Sunday, October 14, 2001

The first indication you're going to have a problem is when the conversation runs something along the lines of "We should throw everything out and rewrite in Java/C#/programming language du jour because it's more reusable!"

But essentially you're right: anything that allows you to distinguish your core business from other peoples' efforts is something you should do yourself.  All too often, people look at their IT budgets, go, "this isn't our core business", and outsource stuff they shouldn't.

If you work in a time critical environment, don't outsource things on your critical path, because EDS will never jump as quickly or as high as your internal staff.  If your core business is financial transactions and your bright inhouse programmers can groom a bespoke system to perform transactions 0.1% cheaper than IBM's outsourcing division, you're nuts to sack them and give the business to IBM, especially if that's what everyone else is doing.

Rodger Donaldson
Sunday, October 14, 2001

This from a man who just wrote a discussion board system.  ;)

I don't the the three questions at the front of your essay support the conclusion -- instead they suffer from being hard and fast rules.  "Code reuse is good" and "Find dependencies and eliminate them" can each be applied as rhetorical devices to support whatever project you do or do not want to pursue.

That said, I think your conclusion is right on the money.  Stated as a business proposition, I would say it: find the systems that create the greatest risks for your business, and find a way to control those risks.

M. Hedlund
Sunday, October 14, 2001

Doing core functions yourself and outsourcing everything else is good business.  Core functions are where you create sustainable competitive advantage, which is where market value comes from.  If you outsource your competitive advantages, they won't be sustainable.

However, this doesn't mean you shouldn't reuse code.  It just means you [probably] shouldn't reuse someone else's code.  After all, the Excel team might have had their own C compiler, but they all used the same one (I assume!)

Ole

Ole Eichhorn
Sunday, October 14, 2001

I think code reuse *is* always good if it provides the greatest exposure for the best code within a given problem domain. This statement is different from "people should always reuse code when possible." I take the "Excel-team-has-its-own-C-compiler" story as evidence of stupidity on Microsoft's part --- not because they rolled their own C compiler, but because no other teams in the company thought to reuse that compiler for their own projects, even though they were aware of its existence and even though it was clearly superior for the reasons you state.

Owen T. Cunningham
Monday, October 15, 2001

I posted a log entry on my own site in response to this, at http://www.platypus.ro/content/entry.php?2001/October/15b.

Jeff Darcy
Monday, October 15, 2001

Given that you run a software company, software developers are at the core of your business. Based on the argument you presented, you should therefore roll-your-own programmers by having sex with several appropriately intelligent women.

The reason this won't work (aside from the various logistical problems) is the timeframe: you'd have to wait at least 20 years until the resource becomes available. And this is exactly the point with any form of code reuse: development time. Simply rolling your code from scratch (machine language on up) takes to much time, even if you have Microsoft's resources.

Obviously your company/products must provide a value add to the customer. But your value add can be built on top of reusable component obtained from an external source. My self, I always groan when I see developers rolling their own C++ containers instead of using STL. They are never better, and nobody else (even on the same team) knows how to use them.

P.S.
I like the new discussion software, but I wish I could see the previous postings while writing my own.

Dan Shappir
Monday, October 15, 2001

I agree totally with the sentiment about rolling your own containers etc.  Some problems have just been solved to well before.

However, I think it is important to look at what you are trying to reuse.  Too often, I hear the cry for "Reuse, reuse" of core business logic components.  These animals are almost always very domain specific beasts, and reuse outside the project does not make too much sense.

It is important to realize that code reuse comes as a cost, both when writing the code, and when figuring out how to use it.  I believe that 95% of what can be reused effectively is already available in standard libraries etc.  What are the chances that the dusty old piece of code tucked into the company's "component library" will save you any time?  What are the chances that you will help someone else save time by "generalizing" your component for reuse?

J. Brodwall
Monday, October 15, 2001

My view (also a developer and trainer):

While I actively endorse software reuse (as is "steal from yourself whenever possible"), there is another side to reuse:

1. If you are near the leading edge, there probably isn't anything else out there to reuse.

2. If there is, it's probably throw-away garbage because nothing is ever given away for free.

Specifcally, ever notice how all the "state of the art"  books only ever rehash the same tired examples?  Not one book EVER (well, at least not until the technology is "old hat"), EVER gives away the real big secrets.

Why?  Well, in the older days it was called "technology transfer".  That is, you could pay a consultant $XX to do the job, but would have to cough up $XX + $XXXXX (lots more) if you wanted them to train your people so that you wouldn't need that consultant.

Bottom line - code reuse is not going to work  as well as hoped in the real world because the good stuff will not be given out with source code (protect intellectual property or  at least all that hard work), and anything without source code is NOT reuse.

Cheers,

-Richard

Richard S. Huntrods
Monday, October 15, 2001

Read Object Lessons by Tom Love, 1993, SIGS Books, ISBN 0-9627477-3-4.

One of the themes of the book is that components are brilliant: someone else spends much more time and effort than you can afford creating a useful component because their payback is over many users.

In my experience this sort of reuse has huge advantages for your development time and budget. The downside is that you don't often find a component that works *exactly* as you want, so you often have adapter code to support the component. Despite this, I usually find that the benefit outweighs the cost.

Jeff Veit
Monday, October 15, 2001

There is one problem that has been missed with using someone else's code, regardless if it is core to the business or not.  You inherit all of their problems.  If you purchase a grid solution (say VSFlexGrid - as an example), and you use it in your solution, your clients don't care one bit that this problem or that problem is actually stemmed from a bug in the grid itself, all they see is a problem in the solution you provided.  When you use your own solution, at least you can control (somewhat) the bugs and errors, and fix them when they come up, instead of being at the tender mercies of the outsourced product.

Brian Copeland
Monday, October 15, 2001

I really enjoyed this article, and the conclusions.  I have particpated as a consultant to many Enterprise projects in C++ and Java where they built their own persistence layer.  It would be useful to write a follow up article about getting re-use in the architecture layer (>50%) of most projects, by using open source s/w.
Regards, Matthew

Matthew Pope
Monday, October 15, 2001

In general, I agree with your axioms concerning code reuse or building it yourself.  It has been my experience that reusing code has the least amount of gain compared to other deliverables built during previous software development cycles.  I have an adage of not reusing code unless the code has been architected for reused and the code reuse actually provides a solution to a problem or requirement (not providing a problem to a solution).

I obtain the greatest amount of reuse from processes, frameworks, patterns, logical designs, and the like (or non-code artifacts).  I see more organizations spend weeks or months working through an implementation when they could have cut that time at least in half by learning the pros and cons from a similar pattern previously published and transforming that similar pattern into a solution for their specific conditions.  Most of us are not innovating protocols, patterns, processes, etc and on some level, someone else in our industry has possibly gone through the pain of a similar problem.  I try to apply my mom’s saying of “learn from other’s mistakes” to software development…if I can only apply that saying to my personal life.

Bob Burdick
Monday, October 15, 2001

NIH is different than "Manage risk".

NIH to me means "we didn't think of it, so it can't be right."

Manage risk means understand where your internal and external risks are, and deal with them in the best way you can. 

From the point of view of the Excel team in the example, this meant "we know from painful experience that we can't count on the compiler team, so let's maintain our own compiler."

There's a cost to that, and if the team would never, as a knee-jerk reaction, look to offload that cost, *then* there's some NIH going on (or maybe they should take over the compiler team :->).

Your conclusion is accurate, however: don't outsource your core competencies.  That's risk management.

The problem is, naturally, identifying risks.  Most people don't do a good job of that. 

I, for example, get extremely nervous about third-party software that I can't build from source in my SCM sandbox.  The risk is that if there's a bug, my project will be at the mercy of someone else.  That doesn't mean I won't use such third party software, it's just a risk.

Take another example: if your area of core comptency is "Y", but before your excellent code implementing "Y" can run, you depend on "X", but no third party can seem to get that right, then you'd be dumb to not *get* a core competence in "X" (or run away quickly).

Risk is just another thing to trade off against cost, schedule, and quality.  I can cobble something together with a bunch of third party components, and that'll work great for a demo, or maybe even for a small deployment.  Or maybe even forever.  But the risks of using third party components just simply weight against, for example, the risk of not getting something done.

NIH is much more dangerous.  It says that the risk of external ideas (or code) is infinite, and the risk of internal ideas is zero.  Very dumb, even if you are smart.

Bill Hofmann
Monday, October 15, 2001

Well, Yes - I'd agree it is many times best to have a walled city.  One that is maintainable and comprehendable and insulates you from the winds of chaos outside.

An interesting example is a lot of work I did several years ago having to up convert old entertainment products from DOS or win3.1 or Mac to win98/NT

Oddly the easiest products to convert were those that were the most self contained.  An example being MortalCombat for windows (converted from DOS) it was so self contained that it even had it's own multi-threading kernel.  Oddly after converting just the kernel and the media driver interface - all the rest just came along for the ride with a bit of editing to get rid of compile errors.

Peter Kennard
Monday, October 15, 2001

It's all about "understanding".  If reusing code means you have to expend effort to understand the code, what it does, how to install it, etc, and that effort is more than it takes to write what you need yourself (cf Joel's case), and maintain it *in perpetuity*, then by all means, write it yourself.

If it doesn't, reuse code to do the function.

That's why SAP is such a bad idea; manyears need to be spent to understand a product so general that you'll only need 10% of the functions, which could be written in a fraction of the time.

Michael Robinson
Monday, October 15, 2001

Of course, this is precisely why online publishers should develop their own content management systems rather than purchasing them from Vignette, Thunderstone or anyone (ahem) else. =)

Ryan Tate
Tuesday, October 16, 2001

Is there any connection between code reuse (or the lack therof) and the fact that I received _three separate copies_ of the "[JoelOnSoftware] In Defense of Not-Invented-Here Syndrome" e-mail message?  I'm guessing that there isn't any such connection at all.  Perhaps it's even my fault.  Maybe there is something that I did that resulted in so many copies.

Anon Lover
Tuesday, October 16, 2001

>Of course, this is precisely why online publishers should >develop their own content management systems rather >than purchasing them from Vignette, Thunderstone or >anyone (ahem) else. =)

>Ryan Tate
>Tuesday, October 16, 2001



It's definitely not the only reason...

http://forum.fuckedcompany.com/fc/phparchives/search.php?search=vignette

Anon Lover
Tuesday, October 16, 2001

Microsoft introduced a very interesting definition of reuse.  As UNIX was busy trying to promote open-source as the ultimate reuse technique ("here are a million lines of code, help yourself"), Microsoft went the binary way and invented COM.

My personal opinion is that it was a fluke.  They had very simple IP concerns in mind and didn't want to open their source, which is perfectly reasonable.  So they focused on binary, and binary interfaces.  And by doing that, they unearthed the only real way that code can be reused:  a clearly defined binary SPI and a set of interfaces.

Why did this work so well?  Because it made programmers more conscious.  Instead of dropping a phonebook of listing on everybody's laps, they had to write code and then pick a carefully chosen subset that they thought might be reusable.  Then they had to document it and export it as an IDL interface.

As a result, the Windows that we know now is the ultimate example of code reuse at work.  HTML component, ftp library, XML parser, you name it.  It's all in there and all it takes is OLEViewer (or whatever its recent incarnation) to browse it and a few lines of VB or C# to put it to work.

The recent CLR is a simple (and very interesting) generalization of this process.

Brilliant.

--
Cedric

Cedric
Tuesday, October 16, 2001

(same post, no html)

Michael Porter wrote an excellent article in HBR a few years back on a similar topic (but not at all about code reuse, of course). I don't know if this is covered in Competitive Strategy (http://www.joelonsoftware.com/stories/storyReader$351) or not, but it's a much quicker read.

He argued that 'operational effectiveness', which is what you get when you use the same system everyone else in the industry is (or will be) using, is not sufficient for success; companies need to figure out what unique value-add they offer and then focus on performing those activities better than anyone else in the industry.

The article is accessible online but unfortunately costs $6. It can be purchased here:

http://www.hbsp.harvard.edu/products/hbr/novdec96/96608.html

There is a free summary of the article (which appears to almost as long as the original) here:

http://www.best-in-class.com/research/bestpracticespotlights/porter.htm

Steve Bronstein
Tuesday, October 16, 2001

When should you reinvent the wheel? When the first inventor made it square.<br>
<br>
(And in my years of software development experience, I've ridden on way too many square wheels.)

Allan Beatty
Tuesday, October 16, 2001

Joel, nice piece.  I've been labelled as a lunatic for saying something similar:

http://www.ericsink.com/weblog.py?cmd=showarticle&article=surprises.html

Programmers certainly don't need to go around reinventing the wheel all the time, but there *are* situations where it is best to build, rather than buy.

Eric W. Sink
Tuesday, October 16, 2001

Bill Hoffman (see above) is absolutely correct.  Build vs buy (aka code reuse) is a moot point.  The driver is (should be) risk management, which is highly subjective and contextual.

Jason Osgood
Tuesday, October 16, 2001

When thinking about reusing software, there is one oft-overlooked point: In order for software to be reusable, reusability must be in its top three design goals.

In other words, if you didn't design with reusability in mind, it isn't reusable.

On the other hand, design for reusability isn't sufficient either. Just as your code isn't correct until you've tested for correctness, and your code isn't performant until you've tested for performance, your code isn't reusable until you've tested for reusability.

This is often overlooked, partly because it's so hard. How do you test for reusability?

1. Find somebody outside your group to reuse your code.
2. Find somebody outside your company to reuse your code.
3. Find somebody of a sort that you weren't thinking about at design time to reuse your code.
4. Fix the problems they encounter and iterate.

The hardest part of all of this is finding someone who is willing to reuse your untested code to accomplish something non-trivial. After all, they have their own risk management to attend to.

So, to squarely (but partially) address the question at the top of this thread (is code reuse the holy grail or a recipe for disaster?): Reusing non-reusable code is almost always a recipe for disaster.

Jim Lyon
Tuesday, October 16, 2001

Thought I'd add my €0.02.

Real-world organic life forms all use this philosophy too -- that of only being dependent on what is absolutely necessary. One branch is cut off a tree doesn't affect another one on the other side.

I think we're going to see more of this kind of model in the resulting software -products- too in the form of greater desynchronisation and more fault tolerance.

Julian Harris
Thursday, October 18, 2001

I read Joel's article with extreme interest since the build or buy question comes up a lot these days. More interestingly, this is a question I faced about 5 months ago.

I was contracted to a large information company north of Atlanta, GA to create a document management system for use over the web. This company had invested in a small company that created document management software they used internally. For reasons I won't discuss here, this large information company got the source code to the document management software. The thinking was to modify the source code, create a client for a web browser and voila! A web based document management service. This was a great way to leverage their investment and sounded perfectly logical to the executives and people working on the project. However, they did not take into account that there was roughly 2 million lines of code with no system design documentation and nobody with any prior experience on the system. Suddenly the thought of reusing the code was a monumental task.

Once it was determined we needed another system to meet the goals, it was a buy or build decision. Ultimately we decided to build the system for the reasons listed below.

1) With around 2 million lines of code and no documentation, I couldn't ensure that my estimates to fix and modify the software were valid. We could have spent our time and budget and be no closer to the goal than when we started.

2) Since the large information company wanted to run this as a service, buying a document management system was simply not an option. If the system went down at 4 a.m. on a Sunday, how soon could a third-party vendor get it up and running? You're probably thinking "Geesh, 4 a.m. on a Sunday - who cares?" Well, in Asia it's around lunchtime on Monday. In the ASP world, there are service level agreements or quality of service agreements (my own term) that penalize the ASP for downtime. Every minute the document management system is down progressively costs more money. Let us not also forget that nobody wants to do business with an ASP that has a bad reputation for service.

3) Resuse of code comes in many different forms. When we built the system, I used the WebDAV standard (an extension to HTTP) and the DoD 5015.2 Rev1 specification (the federal government's specification for document management system) as the blueprints. This saved us 2 months. When I developed the client, I used the W3C's libwww library and modified it to add the WebDAV methods. This saved me about 6 months. I used IIS as the web server and used ISAPI to modify the behavior. This saved me at least a year. Although we were building a new system, we reused a lot of code and were able to implement a full client and 80% of the server before the project got cancelled. Total time: 4.5 months including full documentation.

4) Small development teams are better for about 90% of all software projects. I was the architect/developer/tester throughout the entire project. Some software like Windows 2000 or SAP are going to have very large development teams because they are very complex pieces of software. However, my experience has been that 1 or 2 very high-quality developers are worth about 15-20 average developers. Not only is the system architecture more consistent but you don't have to deal with 10 developers who have a hard time finding their butt with both hands and don't add any value to the team. Find good people and take care of them. Pay them well. Let them have Friday afternoons off to play a round of golf. Trust me, you'll save money and end up with a better piece of software.

Josh Watts
Friday, October 19, 2001

Joel: Very thought provoking article.  Open source development tends to have a very high level of code reuse, it doesn't always work well, but sometimes it works well even when what is `outsourced' is flakey code.  I posted an advogato diary entry on this: http://www.advogato.org/person/chalst/diary.html?start=11

Charles Stewart
Monday, October 22, 2001

I always extend the metaphor of "Don't reinvent the wheel" like this:

There are an ever-increasing proliferation of different kinds of wheels with [as time goes on] increasing fitness-to-purpose: bicycle tires, steel belted radials, monster truck tires, gears, tires for huge mining machines, airplane tires, turbines, etc., etc.  -- not all available components will work for your particular cirmumstances.  A wooden spoke wheel not going to cut it for a tank.  Sometimes, you need to "roll your own" wheel.  heh.

Too many enterprise developers are caught up in trying to follow "correct" processes.  Bah!  I say keep your eyes and ears open, use your brains, and use what works on a case by case basis.  There is an infinite amount of software that can be written - no one process is always going to work.  If you do use a process, make it your own and go into it with a degree of skepticism - make it prove to you that it's working.

If you don't know what you are doing, looking around to see what others are doing is a good idea to get the ball rolling - best practices fall into this category. 

At best, any process or practice you pick up should just be a starting point.  Take what you learn and make it better. 

LWap
Friday, March 05, 2004

What if your software has a longer timeframe than the libraries that you used?  It can end up to be a huge effort to go through the code and rewrite to a new GUI library. 

A smaller library, or a specialized component may be easier to work around, if necessary.

Gary W.
Saturday, July 24, 2004

*  Recent Topics

*  Fog Creek Home