Fog Creek Software
Discussion Board

Daily Build Considered Harmful ?

I wonder if daily builds are still applicable in large software projects (20-30 people). My experience tells me that trying to do a daily build in such systems results in many wasted hours of integration effort every day.

Note that we can automate the build process using scripts etc., but we cannot resolve integration problems automatically.

Another thing - in large software projects it seems logical to separate the project to many sub-projects where each sub-project is performed by a small team (3-4) of developers. It seems much wiser to me if different teams coordinate their integration effort based on stable sub-projects sources rather than their "latest and greatest" version. If this is the case - then it seems that the benefits of the daily build process are much smaller then those described in joel's article.

Am I missing something here ?, or maybe daily builds can be used effectively only when small teams are involved ?

Regarding the Windows NT story - I'm aware of it, what I don't understand is exactly how did they manage to pull this off without getting severe integration problems every working day ... (maybe joel can explain this magic ?)

Saturday, September 14, 2002

The point of daily builds is to force the development team to face the unpleasant task of working together.  If you're having such a difficult time getting things to work together every night, just imagine the problems you'll have to reconcile if you let 30 programmers go their own ways for a couple months.  A couple hours a day is a small price to pay if the alternative is several wasted weeks or even months at the end of the project when no "working version" is in sight.

On the other hand beware of integration efforts which are really just temporary workarounds to bugs in modules yet to be finished.  My last project almost required that we have a bug-free "demo" at every stage of development, so a lot of time was sunk into temporary workarounds.  That's a treadmill you'll never be able to get off.

The sub-projects idea is correct, since the upper bound on the size of any cohesive team is around four or five members.  If you can break down the project into modular sections which have few dependencies on each other, that's best.  In my [relatively limited] experience what works best is to quickly design the app "skeleton" and then having several teams of developers drill down on different parts of the system.

Saturday, September 14, 2002

Hm. I believe the whole point of maintain a daily build is to force programmers to test their buggy code against the main branch.

If you need to maintain several hours doing "integration", maybe it's because your people isn't working "right". Please note that I'm using these terms liberally: I don't see the needing of "integrate" things, you should be working integrally already.

On the other hand, the "working demo" problem can be solved easily using CVS.

Sunday, September 15, 2002

The larger the working group, the higher the possibility of being split into separate teams.

The more separate teams, the higher the possibility of walls (lack of communication) developing between those teams.

The higher the walls, the greater the possibility of limited narrow focus on "specific team responsibilities" and ignoring other teams.

The more narrow the focus, the higher possibility of cracks and things falling into those cracks.

With sufficient cracks and lost things, the integration effort can grow exponentially, consuming more resources than whatever is being changed by the project teams.

I agree with YO... people is not working right.

Joe AA
Sunday, September 15, 2002

Daily build considered harmful?  How about an hourly build?

The goal of a continuous-integration scheme is that no one ever checks in code to the main line that breaks the build.  If a team is working on a disruptive change they do so on an isolated branch, and before they check in to the main branch they integrate tthe latest changes from the main branch to their project branch and build and test the integrated code.  If something goes wrong and a checkin breaks the main line build or test, it's backed out and fixed in an isolated workspace.

This approach means that the overall project never has a prolonged stay in integration hell.
I've seen it work on projects with as many as fifty active developers.  This was for a project written in Java, where builds were fast.

Chuck Karish
Sunday, September 15, 2002

Mozilla has continuous builds. You can even see the status of their build breaks on their web site here:

If they cannot smoothly integrate tiny pieces of code each day, how do you expect them to smoothly huge modules every few months?

Daily builds are great. Just because you have a daily build does not mean that you can't have a private branch where you do messy development work. From the dev branch, you can periodically check it into the main branch when your work is stable.

Zwarm Monkey
Sunday, September 15, 2002

>My experience tells me that trying to do a daily build in such systems results in many wasted hours of integration effort every day.

Why is it 'wasted' effort? Just think of the effort it would take to do the integration after a month or so.

And what about bug fixes etc. in other parts of the software?

We have about 20 developers working on one piece of software (even though it is splitted into many small subprojects) and I would not want to live without a daily build ever again. We even have a daily setup build, so that non developers can get the daily build to their machine to test it and work with it, something that has proven to be very helpful, since bugs introduced with recent changes tend to show up rather quickly.

I cannot imagine why you would need several hours of integration work each day. Don't you guys ever talk to each other (before you check something in? preferably even before you program something?). On a bad day, when we have build errors (about once or twice a week), it takes the person responsible (whoever checked in the problematic code) maybe half an hour to fix it, because normally it is due to a file she forgot to check in or a path that was different on the local machine or something like that.

Have fun,

Jutta Jordans
Monday, September 16, 2002

I don't understand your point. Daily build are the integration build.

Daily build don't mean that you should check-in and build whatever you had time to develop that day, and that's the only build you ever do.

You can work on a specific feature for days and build locally before checkin in to the daily integration build (but after having merged the main tree into your specific changes).

Robert Chevallier
Monday, September 16, 2002

The daily build process is explained in "Microsoft Secrets..." by Kusumano and Phelby - as Robert explained it's not about integration of every programmer's code every day. It's about integrating a logical and working part of a module to the main build - You check in code when it's at a point where it can be integrated. So you ever check in code that won't compile (SCC isn't a backup), you make sure code that gets in is already working with other code in the main branch (in your machine) and you ensure that you have the latest version of the main branch code everyday. In reality a daily build just means you GET source code every day from the main branch - you don't have to CHECK IN code every day.

Deepak Shenoy
Wednesday, September 18, 2002

Some MS VS.NET teams used a system called 'Gauntlet' that accepted a set of files, built them against the current sources,  tested the new build, and, if the files passed, the checkin succeeded and the sources on the gauntlet machines were rebuilt.

50+ programmers checked stuff in weekly without breaking builds--their build break rate was on the order of one break per year.  This explicitly sequenced checkins, so while the initial rate of checkin failures was high, the failures only affected only the individual developer, and the quality of the repository code was extremely high.

Greg Bronner
Thursday, September 19, 2002

Pull a feature branch for a longish development
effort. That branch can be for a smaller number of
people and wouldn't need to have the daily builds.

Friday, September 27, 2002

*  Recent Topics

*  Fog Creek Home