The Old Joel on Software Forum: Part 4 (of 5)

Visual C++ build process

We have a system which takes 5 hours to build in VC++, with no binaries (doing a clean build).
Link time is also a few minutes.
Is there anyway to roughly estimate the number of files present in the code base based on build times, or rather is build time more dependent on coding dependancies upon header files?

Developer
Wednesday, February 25, 2004

You can figure this out pretty easily (kinda). generate a list of all source files (ie all the .obj files), the size of the source (.cpp) in bytes. and then estimate the time taken to compile (if VC++ isnt printing out how long, Borland compilers tell you how long it took to compile the module) then using excel (or whatever) figure out the bytes of source/second. if this ratio is about the same for all files. then you know its based on the sheer number of files that makes your build take so long. if the variance of this ratios is high its bound to the # of header files processed by the CPP. a simple histogram will probably tell you what you need to know. a plus on the histogram is that if its dependency bound, this will tell you what files will give the biggest payoff in fixing dependencies and removing spurious includes.

good luck

if VC++ isnt giving the compilation time for each module, you can estimate it crudely by noting the start time of the build and ordering the .objs based on their update time. the compilation time (is roughly) the time between the last files update time and the current files update time.

mikester
Thursday, February 26, 2004

Genuinely curious. 5 hours for a build? What size project are you working on? The build PC?

I have been working in Delphi for years and anything more than 2 minutes is a huge project. Please don't flame me for this. I am really intrigued.

David

David Freeman
Thursday, February 26, 2004

Isn't there a way of making VC++ show the time taken to compile? Something is tickling the back of my brain, but of course it might just be a bit of fluff that has moved there from my belly button.

Thursday, February 26, 2004

Visual C++.NET 2003 has an option that fires out the build time in the output window.

John C
Thursday, February 26, 2004

Even for a full rebuild, 5 hours seems an incredibly long time -- what sort of project is it?

If you're on a LAN with a reasonable number of machines, take a look at Incredibuild. It's cut our build times down significantly, and the company that makes it have been very helpful about resolving bugs. The best thing about it is that you don't need to have Visual Studio installed on the remote machines.

http://www.xoreax.com/main.htm

Adrian Gilby
Thursday, February 26, 2004

I don't think that you can estimate the number of files strictly from the build time - it's affected way to much by the code in the files, for example you can really tell the difference between projects that make heav use of templates and those that don't.

If you're looking to speed up your builds we found a distributed build system for VC++ 6.0 that really helps - our build time went from just under an hour to about 3-4 minutes. I won't mention any product names because there's a few of them out there and a lot of people on this board already seem to think that I work for one of them. (I don't).

R1ch
Thursday, February 26, 2004

"Isn't there a way of making VC++ show the time taken to compile?"

Add a /Y3 on the command line to start it up.

sgf
Thursday, February 26, 2004

David Freeman,

Our main product (engineering/military analysis app, in VC++) takes 6 hours to compile for a clean and full rebuild, which everybody does nightly (1-2GHZ P4's). It just has hundreds of modules that need to be compiled, each with maybe 50 files.

Ron
Thursday, February 26, 2004

Compile time in VC++ varies GREATLY depending on the code in question.

I had one program that took several minutes to build, while another project with twice as many files and lots more code in general compiled in less than a minute.

The difference? The first program used a lot of templates. The second one didn't.

Chris Tavares
Thursday, February 26, 2004

And correct use of precompiled headers. Check if you use them and the headers included are actually static.
And amount of memory available. Try to add some more memory to the machine and compare build time. Having all header files in file cache may speed up your build.

WildTiger
Thursday, February 26, 2004

> The difference? The first program used a lot of templates. The second one didn't.

I think it helps if your template header files are included in the precompiled header file.

Christopher Wells
Thursday, February 26, 2004

That doesn't help when the templates in question are under active development, it just makes the compiler recompile EVERYTHING more often.

Chris Tavares
Thursday, February 26, 2004

Build a dependency graph and get rid of any unnecessary includes. I worked with a guy about 10 years ago that included stdio.h needed out of mental security in each module he wrote. once he was canned and we cleaned up the source our build was much faster. processing include files often accounts for 50-70% of the compilation time of a module.

optimize your include path. ie if you include mostly from your own headers put that first so that you dont stump through the compiler lib headers needlessly.

If you can dedicate a partition (even 1-2gb) to your .obj files that you clean out each time, you wont have any fragmentation slowing you down when it comes to linking or writing the obj files in the first place.

mikester
Thursday, February 26, 2004

Buy 'Large Scale C++ Software Design' by John Lakos. Apply the principles.

Then buy Incredibuild (as recommended above).

Then upgrade your PCs.

Mr Jack
Friday, February 27, 2004

With 5 hours build time, I recommend reading "Large Scale Software Design" by Lakos - chances are, you'll find one or two eye openers.

What I'd really like to know is why you want to use build time to find out the number of files. There are quite a few other, easier options.

If it's all under one directory, the most simple solution is "dir *.cpp /s". Repeat for .c files, add numbers.

In case that doesn't work, look at the .vcproj and count the instances for '.c"' and '.cpp"' (I really wish there was a <code> or <em> tag, Joel!)

You have multiple projects in there? Fine - do a clean rebuild and take the build log. At least VS.NET creates one - that lists all the files that have been touched.

Hope this helps

Robert 'Groby' Blum
Friday, February 27, 2004

Do you ever notice how when you compile the HD light flashes a lot? I wonder how much of the % of compile time is spent reloading files constantly.

If you can afford it, perhaps buy a solid state drive or card of perhaps 1-4gig, and place all your object files there and perhaps all system include files too, along with your source (and have it synced to a real HD/LAN too)

If you can still afford it, get a dual CPU box with the fastest DDR ram dual channel possible.

If you cannot afford it, I wonder if there are any disk cache programs that can cache all reads on a per folder level instead of everything, and just force it to keep all includes/sources in ram (so add another gig to your box)

And yes as someone else pointed out, do get incredibuild, if you are making projects that take 5-6 hrs to compile, then hell, get incredibuild, buy 5 or 10 rackmount PCs or slim workstation (cheap-ass boxes for $1000 with 1gig ram at 2ghz) and use them all for distributed compiles, as well as anyother not used machines. If your working on military projects then its silly not to do this, you can afford it, which company is so on the edge? that they cannot afford to spend $10k on 10x speed improvements in compile times :)
If you are real cheap, buy 10 1ghz boxes of ebay for $200 each. Seriously, no commercial place should be without it. If you have the cash, quad cpu xeon or amd, which ever your allegience is.

Rally
Sunday, May 30, 2004

I am looking for the answer for my question too...
We have a lot of projects(40) under one single solution. We use the command line (devenv.exe Solution\mysol.sln /build /Release) to build
the solution,
Is that any information I can get the estimation time that will take to build.
My users are compile they don't know how long the time it takes to build, instead of sitting on, and wait for it finish. I am writing some perl script to drive the vs.net build from the commadn line.

Is that a way to find out?

Mav
Friday, June 25, 2004