Fog Creek Software
Discussion Board




Build times

I'm sitting here reading JOS when I perhaps ought to be working because I'm running a build.

It's a build on maybe a medium-sized project (approx 200Kloc). It takes around maybe 45 minutes for a full build, the last time we timed it.

Is this normal? Does everyone really wait this long for things to happen? I know this is part of the rational for nightly builds, but seriously....

I suppose this is just a bitch-post, but gah, at home I have 15Kloc Java projects that compile in 3 seconds, and here you do a build before you leave for lunch. Blech!

Mike Swieton
Monday, May 19, 2003

>It's a build on maybe a medium-sized project
>(approx 200Kloc).

VC++?

Where are you workin', Mike? :-)

I would think you could use modularization and incremental builds to avoid having to do full rebuilds.  You'd only do full rebuilds when you need a version to pass to testing - a "daily build"

45 minutes for 200 KLOC for VC++ sounds about right, for version 6.  I dunno about .net ...

Matt H.
Monday, May 19, 2003

Matt: I'm over at Atomic Object, in Eastown. I need a full build now because I'm not working on code but on the build system and need to see what comes out of it broken.

It's just annoying, because my home box seems faster (at least if I scale the amount of code linearly to match, though I'm not sure if build times are linear...)

I hate QNX 8-}

Mike Swieton
Monday, May 19, 2003

Are the sources local, or are they on a File Server aka Repository.

Do you use precompiled header files

Here th. Ere (e-Very Where)
Monday, May 19, 2003

My system build takes about three hours.  I wish it took an 45 Minutes.  I am reworking the present build so it will not take as long (Per project management).  I build both a Java component and a C++ component. Both components are around 2MLOC (The Java Component is Larger).  It takes the Java part around 45 Minutes and two hours and 15 minutes for the C++ component.  So java and C++ appear (under MS VSC++) are not linear.

A Software Build Guy
Monday, May 19, 2003

Boy, I don't miss C++ build times. :)

Our C# multi-project solution is ~2k files, ~100kloc. Full rebuild time is under 10 seconds.

Brad Wilson (dotnetguy.techieswithcats.com)
Monday, May 19, 2003

I have ~40Mb source in 4000 files, and it takes less than 10 minutes on a _really_ old machine, (Visual C++ 6), to compile the whole thing. What's different? (could be it that this is made of many small sub-projects with relatively few interdependencies?)

If you have a lot of RAM, try this, (I used to)
1. Make a huge RAM drive
2. At start of build script, copy entire source tree to the RAM drive
3. Build it on the RAM drive
4. Copy the results back to the output directory on the real drive

S. Tanna
Monday, May 19, 2003

This sounds like a dodgy plug, but I promise that I'm not related to the company in any way whatsoever.  We recently discovered a product called 'Incredibuild'.  I have to admit that I was put off by it's kind of crap name and dubious claims at first but we gave it a go and it really works!  It's a Microsoft Visual C++ addin.  The idea is that you get lots of people to install it and then it distributes the build work between each machine.  It seems to work using some kind of virtual machine system, so you don't have to have the same configuration on each machine - in fact it says that you don't even have to have Visual Studio installed on all of the boxes.  We had a few issues at first because the dependencies weren't quite set up properly in our workspaces but it is working excellently now - a clean build used to take just under an hour (on a 2.4GHz Pentium),  but my last build took exactly 3 minutes and 12 seconds - needless to say, I love it!  On the downside, it's pretty expensive, but I was surprised how easy it was to convince the bosses when I compared it to man hour costs.

R1ch
Monday, May 19, 2003

Oops - I guess an URL would help... http://www.xoreax.com/main.htm

R1ch
Monday, May 19, 2003

Big C++ project in VC 6 a while back, full build - 9 hours.

punter
Monday, May 19, 2003

>> 45 minutes for 200 KLOC for VC++ sounds about right, for version 6.  I dunno about .net ... <<

Is this a joke??  I just did a full clean build of a ~250 KLOC project with VC++ 6 on a relatively slow machine in 8 minutes.  And there have been no attempts to optimize build times for this project.

SomeBody
Monday, May 19, 2003

Our current build process takes around 2 hours (build machine is a bit elderly), but on my laptop (1.7Ghz 256Mb) it takes around an hour or so.
The final output is around 400Mb of executable, source code is 55Mb raw, but this is shared multiple times to give around 550Mb of compilable code.
Currently most of this is in VB and we do use VB groups to speed things up (saves on compiler load time) Since no-one else uses the build machine it's never been a big problem.

Peter Ibbotson
Monday, May 19, 2003

My current MSVC6 project is about 200,000 lines of code - takes about 18 minutes to build (both debug an release)...

Spaghetti Rustler
Monday, May 19, 2003

As an earlier poster suggested, are you using pre-compiled headers? If not, you should be. You should have the includes organised to assist the include optimisation.

Also, these massive builds are ridiculous. You should componetise the architecture so that no no build is longer than 5 minutes.

echidna
Monday, May 19, 2003

That should be componentise.

echidna
Monday, May 19, 2003

Its a while since I used QNX, but it isn't really comparable to a Visual C++ type build.

Plus, as Mike Swieton said, he's testing a full system build, that means everything gets built from scratch.

Simon Lucy
Monday, May 19, 2003

I was really suss about Incredibuild as well until we tried it.

It rocks! It even sped up build times on a single machine (5-10%).

Now we just have to get funding... :-(

Taka
Monday, May 19, 2003

It could be worse, I remember the good old days when a compile could take a day to get done, depending on the mainframe load.

John McQuilling
Monday, May 19, 2003

Have you looked at a copy of "Large Scale C++ Program Design" by Lakos? It has a lot of tips on reducing build times. (e.g. reducing dependencies, eliminating redundant file reads, etc.)

Devil's Advocate
Monday, May 19, 2003

"This sounds like a dodgy plug, but I promise that I'm not related to the company in any way whatsoever.  We recently discovered a product called 'Incredibuild'.  I have to admit that I was put off by it's kind of crap name and dubious claims at first but we gave it a go and it really works!  "


I have seen sooooo many spam emails start off this way....cant we keep these morons out of the forum?

FullNameRequired
Tuesday, May 20, 2003

>>> I have seen sooooo many spam emails start off this way....cant we keep these morons out of the forum?

How about disallowing anonymous posts :-)

Sometimes it actually does happen that someone finds a product that is really good and would like to share that information with others. It makes a nice change from slagging Microsoft products :-)

Taka
Tuesday, May 20, 2003

Hey, I'm sorry - next time I won't bother.

R1ch
Tuesday, May 20, 2003

ye gods.....spammer tag teams...



"Sometimes it actually does happen that someone finds a product that is really good and would like to share that information with others"

I have no doubt.  and yet, in all the time Ive had friends and strangers genuinely say nice things about software, they have *never* felt the need to say things like 'it really works!' (note the exclamation mark).

or..."at first I was doubtful but then it turned out to be really great and solved all my problems."

come on..if you must spam to the lists like this like this at least learn some new lines..

FullNameRequired
Tuesday, May 20, 2003

I don't know why I'm bothering with this, but it does offend me to when you call me a spammer  (moron, I can accept ;).  I'm actually a lead dev for a company that develops a product that has been slagged off a lot lately on this forum.  I've kept out of the discussion as I have no objection to people slagging off our products - I'm sure that they have good reasons, and unless there's something specific that I can help with, I'm not going to join in the discussion - I don't want to offer a biased opinion.  Funnily enough, I also thought about posting something about Incredibuild to the 'software that rocks my world'-type thread, but I didn't because I thought someone would probably get on their high horse about spamming.  Don't get me wrong - I hate spamming too, but that doesn't mean that people shouldn't try to help others out by pointing out good stuff they've found.  If I don't beleive them then that's fine, but I wouldn't reply and slag them off - just maybe they're telling the truth.  If you want proof of who I am then just click on the mail link below and I'll mail you back from my work address, but I suspect that it's more likely that you'd rather assume I'm a liar.

R1ch
Tuesday, May 20, 2003

Hi Rich,

"I'm actually a lead dev for a company that develops a product that has been slagged off a lot lately on this forum"

??? ...and that would be...? 
<g> Im really bad at mystery meat competitions...


"I hate spamming too, but that doesn't mean that people shouldn't try to help others out by pointing out good stuff they've found"

I agree.  But there *is* a difference between such posts and posts that are spam. 
Your post was (deliberately I have no doubt) spam.
I have an email client that uses the bayesian thingie to separate spam out from the normal emails. 
Just out of interest I emailed myself your post.
<g> proves nothing whatsoever of course but it was a pleasant way to spend a few minutes.


"If you want proof of who I am then just click on the mail link below and I'll mail you back from my work address, but I suspect that it's more likely that you'd rather assume I'm a liar"

_much_ more likely.
The problem we have here of course is that I *do* believe you are a liar.  Which means that providing you any personal information whatsoever about myself is a stupid thing to do.
You could easily break the deadlock though, if your product is a good one and you feel no shame about standing behind it then by all means post the url to it, and maybe just a little info about where you fit in    :)

FullNameRequired
Tuesday, May 20, 2003

OpenMosix.  If your compiling on Linux etc, it makes a big difference (if you have a large pool of otherwise idle machines, e.g. workmates).

I am in no way associated with OpenMosix.  Just saw it was a bit cheaper than incredulousbuild(tm)

Nice
Tuesday, May 20, 2003

Fair enough (although I never normally like to publish identifying info on the web either).  I work for Crystal Decisions - the makers of Crystal Reports.  I don't actually work on Crystal Reports though - I work on the Crystal Analysis line of products (  http://www.crystaldecisions.com/products/crystalanalysispro/default.asp ).  I used to work on the server side of their zero client - a DHTML interface that allows OLAP analysis with nothing more than a web browser installed, but now I'm working on a new product that isn't available yet.

BTW, at the risk of being flamed for plugging again, http://www.spamgourmet.com is a great way of being able to make up disposable email addresses on the spot so that you don't have to worry about who you give them to.

R1ch
Tuesday, May 20, 2003

Hi Rich,


and thus I am proved a pratt ;)

Please accept my sincere apologies for the totally unfounded attack on your integrity.

spamgourmet does look pretty good too  <g>  all I have to do now is overcome my reluctance to give them my email address....maybe I could use a second throwaway address..

FullNameRequired
Tuesday, May 20, 2003

No problem - I admit that my message did seem spam like, but I reckon that I've probably gained back at least an hour a day, so I do tend to rave about it.  ;-)

R1ch
Tuesday, May 20, 2003

I work on a 75K line application. It is very heavily templated C++, using a lot of STL, and a lot of non-templated STL.

We find it takes ~ 45 minutes to compile on Visual Studio.Net, and a bit longer (65 minutes) on GCC 3.2

I think that the compile time depends on exactly how hard you work the compiler - there is another group here we interact with a lot - their 35K line c++ app (which is mainly straight C) does a full recompile in about 2 minutes!

regards,
treefrog

treefrog
Tuesday, May 20, 2003

Just checked. My current Delphi project of 236K lines of code fully builds in about 9 seconds. Not that I do a full build often. I generally hit the Run command, which compiles changes only, links and runs the application in less than 5 seconds.

I don't envy you C++ guys for compilation times.

Jan Derk
Tuesday, May 20, 2003

Another vote for Incredibuild - what a fantastic bit of kit!

I would also strongly back that recommendation for Large Scale C++ Design, we were able to half the time of our 300kloc editor from 16 mins to 7 and a half with a little re-organisation and the use of external header guards.

Mr Jack
Tuesday, May 20, 2003

One way to improve the build times on VC++

1. Disable .bsc files creation from Project Settings. Uncheck 'Generate Browse Info' from C++ Tab and 'Build Browse Info' from 'Browser Info' tab.  In my experience this typicaly reduces the build time by 50 %

2. Also try disabling the PreCompiled Headers. In one of my projects disabling 'precompiled' headers actually improved the build times.

3. Check your includes. If there are many include files or recursive includes compilation time unneccessarily increase. Even if you are using header file guard (#ifndef/#define/#endif) the compiler still has to  read the entire file to reach #endif and discard the file.

Nitin Bhide
Tuesday, May 20, 2003

I'm with Jan - I love Delphi compile times. I work on a set of smaller apps and I turn on "show compiler progress" because without the feedback it's so fast that I worry I might have missed the button or something :).

Ryan Eibling
Tuesday, May 20, 2003

This is obvious stuff, but FWIW:

1. Have file guards on all your headers

2. List them in the same order in each C++ file, or better yet pull most of them from a common include file

3. Use pre-compiled headers

4. Avoid multiple inheritance, templating, excessive forward references

echidna
Tuesday, May 20, 2003

How odd, I was just speaking to a fellow developer (form another company) about build times. I forget the exact number, but his build was taking around 8 hours to complete.

Want to guess what he told me solved it? Incredibuild. I'm not sure what flavor Kool-Aid they send with their product, but he was a walking advertisement for them.

Marc
Tuesday, May 20, 2003

Dear Full Name Required,
                                      Why are you apologizing. Why have you given up on your belief that the giuy is a liar. Because he's given you a link to a web page. If I said I really worked for Bush and gave you link to the White House web page would you believe me?

                                      The guy is obviously a spammer, and is simply using other aliases to create the tag spamming effect.

Stephen Jones
Tuesday, May 20, 2003

People should stop being assholes when others are trying to help, or people are going to stop helping.

Funny, virtually this exact same thread transpired a few weeks ago on a mailing list I was on. It's rather shocking to see how radically different the reaction was.

Brad Wilson (dotnetguy.techieswithcats.com)
Tuesday, May 20, 2003

Hi Stephen,

" The guy is obviously a spammer, and is simply using other aliases to create the tag spamming effect. "

<shrug>  you may be right. 
I am persuaded that he is not a deliberate spammer. 

FullNameRequired
Tuesday, May 20, 2003

Rich genuinely works at crystaldecisions.com or at least can send mail from an address there, and I can't seem to find any link between him and xoreax.com which is registered in Israel, so I must accept that he is not a spammer; he merely gives a perfect imitiation of one :)

Possibly we are all receiving so much spam in our inboxes that we will end up talking just like spammers do.

Stephen Jones
Tuesday, May 20, 2003

Even if "R1ch" did have some relationship with Incredibuild other than that of enthusiastic user (and I don't believe he does), what is wrong with him posting an on topic reply to a problem with build times?

It wasn't junk mail, in that he responded personally to a request for information.

His reply was on topic, and backed up by other correspondents with apparently similar experiences.

Not that it is ever likely to happen in this forum, but if someone posted a message for information about electronic banking systems or hybrid rule based expert systems tools I would reply in a similar manner about our own software.

But on the topic, I'm astonished at the long build times for reported from VC6. We use this as our development platform for Windows-based delivery. I just ran a build of our largest component, which has just under 350,000 lines of code, and it compiled in less than four minutes. That was on an old battery-powered notebook with a slow hard disk, 128M RAM and WindowsME, and a 500Mhz processor.

Could the difference be entirely due to templates and stuff, which we don't use?

HeWhoMustBeConfused
Tuesday, May 20, 2003

"Could the difference be entirely due to templates and stuff, which we don't use?"

In general, yes.  Rebuilding a project that relies on STL types will bring the fastest system to its knees. The other big timesink is the Windows header file collection, although precompiled headers can speed this up.

Chris Nahr
Wednesday, May 21, 2003

I'm working under Linux, and I've found ccache to be very helpful for our system - http://dp.samba.org/ccache/ . We obviously don't have the option of pre-compiled headers (which is too bad, as our code tends to include every header in the system!), and we have a massively irritating class heirarchy (including a lot of wierd templating, massive intertwining of classes, and darn near everything descending from a class named Object. Don't ask why - it makes my head hurt.). I used ccache to take us from a >2 hour full rebuild time down to about 25 minutes. I also used it (with a shared cache) to reduce rebuild times by sharing the cache (sort of a cheap version of what ClearCase could do). I don't know if there is any sane way to integrate ccache with VC++ (I suspect it would be hard, but perhaps possible), but as it's free, it might be worth looking at.

Michael Kohne
Wednesday, May 21, 2003

The code I'm working with uses things like templates and STL and includes Windows headers fairly frequently.  I'm not seeing exhorbant build times (8 minutes, ~250k lines, relatively slow machine). 

I'm curious if there might be a correlation between modularizing and build times?  The code I'm working with is broken into quite a few DLLs. 

Are those of you with ridiculously slow build times running virus scanners?  If you are, you might want to try temporarily disabling any active scan features while building and see if it makes a difference.  I've found that virus scanners can increase build times greatly (since they seem to grab onto files whenever they are saved or loaded).

SomeBody
Wednesday, May 21, 2003

Just to toss more weight on the pile, we purchased a bunch of Incredibuild licenses around a year ago and I consider it some of the best money we could've ever spent.  Fantastic investment.

Jeremy Statz
Friday, May 23, 2003

I dropped'em like a bad habit, -- don't put build stuff on my resume any longer.

Steve
Friday, May 23, 2003

*  Recent Topics

*  Fog Creek Home