Fog Creek Software
Discussion Board




Yahoo Stores rewritten from Lisp to C++ and Perl

Paul Graham likes to brag about the efficiency of Lisp programmers. His online store software was written in Lisp and then sold to Yahoo as Yahoo Stores. Fearing Lisp, Yahoo's programmers have rewritten Yahoo Stores in C++ and Perl.

http://www.ai.mit.edu/~gregs/ll1-discuss-archive-html/msg02367.html

In Paul Graham's words:

(a) The reason they rewrote it was entirely that the
current engineers didn't understand Lisp and were too
afraid to learn it.

(b) The resulting program is a new world's record case
of Greenspun's Tenth Rule.  The Yahoo Store Editor called
compile at runtime on s-expressions made on the fly.
To translate this into C++ they literally had to write a
Lisp interpreter.

(c) Even then, they had to drop some features (involving
advanced uses of closures).

runtime
Wednesday, February 26, 2003

Software = programmers + code + users

In Yahoo Stores, the key ingredients were Paul Graham + LISP. When Paul & team were no longer involved, it made sense for management to explore the most effective replacement combo.

It's not a loss/victory for anyone involved. The proof of the pudding will be in the tasing. Personally, I'd take Paul + LISP as the engine any day over alternatives, but I'd be equally alarmed if Paul departs for greener pastures.

If this topic is simply a language troll, I can play that. Switching to C++ and Perl would be like rewriting the code from scratch, so I think it would be very high risk. Especially from a design perspective-- those langs put the hog in orthogonal.

Edoc
Wednesday, February 26, 2003

I maintained Lisp code for a fairly large simulation system back in the late 80's.  I don't know how cleanly Paul and associates wrote their code, but some of the stuff I had to maintain was too awful for words.  The hackerbastuds that wrote the original system had code writing code writing code, with the end result that I had no way to know what effect a given function would really have on the system without actually running the bloody thing many times and watching what happened.  I still wake up panting and sweating in the middle of the night over it.  In the words of Allen Hollub, "...an unholy mess...."

I'm sure it's possible to write robust, clean code in Lisp, but there is an enormous potential there for abuse. Unfortunately, the authors of my simulation system capitalized it on vigorously, probably because it was just so easy.  Or maybe I was just a dumb-dumb,I don't know.  Either way, though, that experience has made me wary of large Lisp systems, and at least in that instance, I think the TCO numbers would have benefitted from a rewrite into something more predictable and straightforward.

I will say, though, that I do miss some of the features I enjoyed with Lisp, real closures not least among them.  When you need 'em, you *really* need 'em.  Maybe if Lisp.NET meterializes, you could use that stuff exactly where you need it, and stick to something a little less abuse-prone for everything else.

Recovering Lisper
Wednesday, February 26, 2003

I underthand that people theem to like it better now.  That it ith written in thee pluth pluth inthtead of lithp.

Mike
Wednesday, February 26, 2003

As JAPH (and a webdesigner who has used Yahoo!Store for the past two years) who likes to tinker with CLISP/Scheme for fun and education, I have to say I take offense at Graham when he says, "The reason they rewrote it was entirely that the current engineers didn't understand Lisp and were too afraid to learn it."  Not "offense" since I'm not a offended, but my brain is tired.  What word means that I think a lot less of him because he's shown himself to be little more than a Lisp fanatic?  Anywhen...  Am I crazy to think that if Lisp were a key feature to running an online site, like Yahoo!Store, they'd have hired Lisp hackers instead of porting it over.  (Geez, obviously it was cheaper to port it, or they would have hired Lisp hackers.)  Maybe he was being tongue-in-cheek, maybe (okay, definitely) I'm  Perl biased, or maybe I'm just punchy right now; but saying that Yahoo! had to port this because their coders were afraid to learn Lisp sounds awfully...  Dumb?

This is one reason I really, really appreciate Joel.  It's also one thing I like to say I've really, really learned from reading his articles.  You use the most cost-effective(1) tool for the job.  If Lisp got Viaweb off the ground and put them ahead of the game, then that was the right tool for the job.  If Perl and C++ are the most cost-effective tools, then the same rule applies.  I don't think fear was what kept them from learning/implementing Lisp, it was (I feel) the concept return on investment(2) that helped make the decision.

On the upshot, I guess this explains what that new store interface thing was all about.  :)

---

1. My brain is very weary, so I'm mangling terms.  By "cost-effective" I mean to include such variables as: what makes the customer happy, what runs the best, what can be developed in the least amount of time.  Whatever could be used to streamline the project and increase shareholder wealth(3).

2. Return on investment in the sense that they'd make/save more money by porting the code over to Perl and C++ than they would training their coders to use/write Lisp.  Sounds reasonable to me anyway.

3. Yahoo! is a public stock, which means they do have shareholders.  If they have any managers their with Business Administration degrees, and if their schools were anything like mine, then the first lesson that had beaten into their skulls was: "Increase Shareholder Wealth." :)

Andrew Burton
Wednesday, February 26, 2003

"This is one reason I really, really appreciate Joel.  It's also one thing I like to say I've really, really learned from reading his articles."

Isn't one of his articles all about the boneheadedness of rewriting working code from scratch?

"Am I crazy to think that if Lisp were a key feature to running an online site, like Yahoo!Store, they'd have hired Lisp hackers instead of porting it over."

In a follow-up message, Graham wrote the following:

"A friend of mine (who's probably on this list, actually) went to interview there in about 2000. He said they told him they didn't need Lisp hackers, because they were going to port the Store Editor to C++. Why? Because they didn't think they'd be able to find Lisp hackers."

So running down the cost-effective list:

"Making the customer happy..."  Graham comments again:

"The feature they had to take out because of closures is actually a fairly important one, so I suspect many users never will upgrade."

"Runs the best..." Up in the air at this point.

"Developed in the least amount of time..." Two-and-a-half years vs. a system that's already running.

Regarding the return on investment, there seem to be three possible equations:

Incremental costs of hiring new CL programmers as existing ones leave (counting Graham et al. as working for Yahoo for a while).  There would be N on staff, doing upgrades.

Incremental costs of hiring new non-CL programmers as replacements, and training them in CL (even if they're already in-house, they count; there will be missing slots to fill).  How much *does* it cost to train people to write Common Lisp?  Assume that there would be P on staff, more than N to allow for more mentoring.

Incremental costs of hiring new Perl/C++ programmers to spend 2.5 years to duplicate (most of) the functionality.  Assume that there would be Q on staff.

Graham and others would probably assert that N < P < Q.

So what ends up costing more?

rwh
Wednesday, February 26, 2003

>>> (a) The reason they rewrote it was entirely that the current engineers didn't understand Lisp and were too afraid to learn it. <<<

I can't imagine the engineers being "afraid to learn it".  More likely they'd jump at the chance to learn it.  This sounds more like management being afraid of the risks of maintaining a language that was unfamiliar to them.

mackinac
Wednesday, February 26, 2003

"Maybe if Lisp.NET meterializes, you could use that stuff exactly where you need it, and stick to something a little less abuse-prone for everything else. "

It may have already materialized. Check out DotLisp ( http://www.richhickey.com/dotlisp.htm )

Ken Dyck
Wednesday, February 26, 2003

rwh, maybe I'm missing something in your post, but I have to ask: If Yahoo! was porting to Perl/C++ because they didn't think they could find Lisp programmers, then how is it considered cheaper for them to keep hiring Lisp programmers that they can't find?  To borrow your quote, Paul Graham did say, "Why? Because they didn't think they'd be able to find Lisp hackers."  If Yahoo! thought that they wouldn't be able to find Lisp programmers -- and, yeah, that does sound stupid, I grant you that -- then rewriting from scratch in Perl/C++ was their best viable option if they wanted to be able to keep running and upgrading Yahoo!Stores five or ten years from now.  I agree with you and Joel, it's generally a bad idea to start from scratch.  However, if you're working from the belief that "Lisp hackers died out twenty years ago when a huge asteroid his the Internet," then you're left with two options (as I see it):

1. Teach your programmers to program in Lisp, starting them off in a new language that is as different from C as Mandarin is from English.

2. Get your programmers to start working on the project, from scratch, in a language they're already familiar with so that it will be serviceable in five or ten years.

I do want to clear up one thing.  My post probably sounded like I was trying to defend Yahoo!'s decision, but I'm not.  My main point was that I think rewriting the code in Perl/C++ was a managerial decision based on their own projections (erroneous or otherwise); it wasn't because their Perl/C++ hackers were scared of a new language.

Andrew Burton
Thursday, February 27, 2003

Recovering Lisper wrote:
"The hackerbastuds that wrote the original system had code writing code writing code, with the end result that I had no way to know what effect a given function would really have on the system without actually running the bloody thing many times and watching what happened. "

Strange. According to what I understood from Paul Graham's articles, it's specifically this capacity of Lisp macro to create program writing program (the Lisp macro) which is particularly interesting.

It is in your opinion this particular program which was badly designed, or the general dark side of this feature appearing during maintenance?

R Chevallier
Thursday, February 27, 2003

The program-that-writes-programs capability of Lisp is very interesting and extremely powerful.  Unfortunately, it is also very easy to abuse.  Without this capability, I can easily look inside a function and get a pretty good idea about what the function is doing and how it is altering the state of the system.  When I allow the function to write other functions, though, it's hard to determine the effects of calling the function without looking at the details of the other functions that are being generated on the fly.

Another problem is with encapsulation.  If I make a change local to a C function, it's usually safe to assume that the effects of that change will be local.  With the code generation approach, though, the change may affect the functions which are generated and functions which those generate, and so on.  Trying to puzzle that kind of thing out is really hard on the molars.

Certainly, a lot of the problems I had stemmed from extremely poor documentation on the part of the original developers.  There's a strong case to be made that I should never have to look inside a function to be able to understand its observable effects on the system.  In my case, though, this wasn't close to being true.

Even more problematic, though, is knowing how to safely evolve a function like this.  Documentation might help with this, but only marginally.  Changing behavior virtually requires you to look inside the system you are changing.

Finally, these folks apparently got off on using this kind of approach whenever they could.  They employed this technique in lots of places where a simpler solution would have worked just as well, IMHO.  The technique is very analogous to the use of indirection - there is definitely such a thing as overdoing it.

Like Spiderman says, "...with great power, comes great responsibility...."  These folks were just plain irresponsible.  Could be that proper controls (reviews, standards, whatever) might help to diminish this problem, but the potential is definitely there.

Recovering Lisper
Thursday, February 27, 2003

Ahh...  That description takes me back to the days of flame wars over C++ operator overloading.

I'll agree with Andrew Burton that Yahoo's decision was a reasonable response to a perceived crisis.

rwh
Thursday, February 27, 2003

How is this any worse than C++'s template metaprogramming?  (Or those crazy text macros.)

Thing is though that companies that don't specialize in software will to some extent rely on lingua francas.  Lisp is not one.

anon
Thursday, February 27, 2003

I've never been to it, but I'm guessing something called "Yahoo stores" could have been written in any damn language. What complicated stuff does it need to do, show me a list of products?

xyz
Thursday, February 27, 2003

This isn't necessarily different than C++'s template feature.

Note how often people use C++ templates in real life.  They've caused so much confusion and madness that people still put on rubber gloves before using them.

Brent P. Newhall
Thursday, February 27, 2003

Hello! Templates and lisps ability to create and redefine functions at run time are not the same thing at all.

Charlotte C.
Thursday, February 27, 2003

Macros are not about creating functions at runtime, as I understand.  It does it at compiletime.  In fact, you can use the functions macroexpand and macroexpand-1 to see what macros turn into.

I really like common lisp because there are all sorts of utils built into the standard that really soften the corners.  Like the disassemble function, which you pass a function to and it prints out the function's assembly code.  Really nice when you're perturbing a function to optimize, and you just ask a lisp listener what it disassembles to.

I can only wonder how a lisp person views how painful programming in C++ or Perl is... ;P 

sammy
Thursday, February 27, 2003

OIC.  Recovering Lisper was probably not talking about macros.  Must be tired.

sammy
Thursday, February 27, 2003

Wasn't this predictable?
Yahoo bought the market leading product and is now in the process of Yahooing it...

I would predict the following consequences(all numbers are guesses):
1. 20 lispers reduced to 0.
2. 20 c++ programmers swelling to 200++.
3. Time for new feature development going from days to months to years to never.
4. New Attitude: It sort of works doesnt it, why change it?
5. Conversion from PROFIT center to net LOSS center.

Net result:
Yahoo Stores is just another typical package.

Steve Burnett
Monday, March 03, 2003

"I will say, though, that I do miss some of the features I enjoyed with Lisp, real closures not least among them.  When you need 'em, you *really* need 'em.  Maybe if Lisp.NET meterializes, you could use that stuff exactly where you need it, and stick to something a little less abuse-prone for everything else. "

hate to be a broken record here but,

"um, javascript?".

mr. monkey
Wednesday, February 25, 2004

There's far too much here to reply to, so I'll just reply to Joel's
own comments.  I am not entirely up on the current state of available
Lisp implementations, but I do know something about Lisp.  I wrote the
first Lisp-coded Emacs ever, I was one of the "Gang of Five"
coordinators of the design of Common Lisp, and I was one of the
co-founders of Symbolics.

I have built, in Lisp, not only a powerful Emacs-class editor, but
also an object-oriented database system, an implementation of a
predecessor of TCP/IP, an interactive debugger, and an instant message
facility (lots more but it's been so long that I don't remember any
more).  My colleagues built a window system, an online help system, a
distributed naming/directory system, a "make"-type facility, superb
compilers and syntax-directed editors and debuggers for full standard
Fortran, Pascal, and C, an extremely powerful mail client, a mail
transfer agent, a powerful user-interfaced toolkit, a high-performance
file system, etc, etc.

When choosing a language for a real-world project, the inherent
properties of the language (e.g. does it have real closures) usually
are, and should be, secondary to many practical considerations.  Will
we be able to hire programmers who know the language?  And if not,
will programmers be scared off by the language (e.g. by an unfamiliar
syntax), whether their reasons be "good" or "bad"?  Are there
implementations that are very well-debugged, highly optimized,
available on all relevant platforms, and free?

Chris's original point about availability of libraries is particularly
valid and germane.  And not only "are the libraries available" but are
they available on all platforms and will they continue to be upgraded
as the computing ecosystem upgrades, to handle new revisions of
underlying resources such as hardware, network protocols, window
systems, etc?

I take no position on whether these criteria are or are not true for
Lisp right now.  My point is about which criteria are the appropriate
ones to consider.

Programming language acceptance is a positive-feedback phenomenon, a
snowball effect, a bandwagon effect.  There are lots of people out
there writing libraries for, say, Java, and making fast Java
implementations, and they are putting those efforts into Java because
there are lot of Java users, not because of which programming language
features Java has versus which Lisp has.  And the result of their
efforts is that there are more Java users.

Joel, saying that programmers have "voted with their feet" is
unrealistic.  99.9% of them didn't get a vote to begin with.  Yes,
millions have learned Lisp and do not use it, but is it really because
someone gave them the opportunity to choose which language to use?  I
currently work for BEA, and when a new programmer at BEA is hired,
nobody asks him to choose which language to write in.  And the same is
true for someone joining the Linux kernel effort or joining the
OpenOffice development effort.  Your world of people "choosing" not to
use Lisp is some other planet than the one I live on.  By the way,
that includes me: I have not programmed in Lisp since I left Symbolics
in 1988.

Are there productivity gains to be had from Lisp as compared to other
GC'ed languages?  I could argue either side of that one, but the most
important point is that this just doesn't matter very much compared to
the above practical concerns.

Joel's claim that in order to write fast code in Lisp, you must avoid
"cons", is exactly as valid as the claim that in order to write fast
code in Java or C#, you must avoid "new".  The two scenarios are
exactly analogous.

If I started up my very own major software project now, I would make
my language decision based on the criteria above.  Sure, I love Lisp,
CLOS is awesome, real lexical scoping rocks, Lisp macros are way cool
(when used as directed), and personally I really like Lisp syntax.  I
am rooting for ITA Software, and hope for and expect them to continue
to greater and greater success; more power to them, by all means.  But
it would take a lot, or require special circumstances, to persuade me
to choose Lisp for a major software project, if I were in charge of
making the choice.

Dan Weinreb
Saturday, February 28, 2004

One problem I often find is that development environments, especially those created by the development project, usually forget about "those who will follow."

Someone eventually has to take over an existing system, either because of staff turnover, acquisitions and mergers, or they got desparate enough to bring in a contractor. When this happens, most systems are not documented well enough for even someone with superior skills to learn about the system in a reasonable amount of time. It takes so long to figure out how not to make things worse, its faster, simpler and easier to do a complete rewrite.

Back in the days of batch JCL and COBOL programs, it was relatively easy to slog though a system and figure it out by following the stream of execution. Even when you had CICS transactions it wasn't too bad. But on modern systems with event-driven processing and messages passing between multiple distributed objects, its very difficult to figure out what's going on; and your printouts of the system are just a bunch of disorganized functions. This is especially problematic for environments like Lisp and SmallTalk (and even Visual Basic) that allow you to "grow" the system over time, without maintaining a set of source code files. Often functions show up in printout based on when they were added to the system.

I once read a classic paper on "rationalized design" as the desirable document to leave behind. This explained the design and the reasons it was designed that way. However, it was "rationalized" to make it sound like that was how they always meant to be, not the dirty and convoluted process that lead to the final design.

David Lathrop
Sunday, March 07, 2004

Maybe people should try programming in Python. It is one of the very few languages which is suitable for both quick prototyping and programming in the large and at the same time interfacing well with C based libraries.
<http://www.python.org>

Hemanth
Monday, March 15, 2004

More discussions about Lisp

http://c2.com/cgi/wiki?WhyWeHateLisp
http://c2.com/cgi/wiki?LispSucks

Jorry T
Wednesday, July 07, 2004

*  Recent Topics

*  Fog Creek Home