Fog Creek Software
Discussion Board




Daily Builds are your friend... but using what ?

I would be interested to hear what tools / packages / kludges people are using to perform automated software builds.

We are looking to automate our build process - approx 30 C++ dll's, 90 VB dll's with a complex set of relationships between them so we need to be able to specify the order that things are built in, and also perform tasks such as dll registration during the build process

TIA

Matt Trinder
Wednesday, November 14, 2001

I don't know about VB--I'm not aware of any way to build them without opening VB and hitting the compile button--but VC projects can easily be built from the command line.  You can control this from a batch file (watching for errors) or from a perl script if you need to get fancier.

There's an article in MSDN about command line compiling at

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vcug98/html/_asug_building_a_project_from_the_command_line.asp

Chris Dunford
Wednesday, November 14, 2001

You can do a lot with the Scheduler built into Windows, and a batch file.  But it's still a batch file, will all of the annoyances that come with that.

For a nice, GUI-driven build program, I've come to like Visual Build from Kinook Software (www.kinook.com).  I kick it off with the Scheduler every night.

And where it shines is that since it is simpler to maintain and manipulate than a batchfile, I've had more time to add yet more automation into the build process.  Which makes my life easier, and gives me more time to do more important work.

John
Wednesday, November 14, 2001

You can build VB6 projects from the command line by calling:

vb6.exe <project.vbp> /make

One simple way to do auto-building (simple usually works best for me) is to write a small app that performs each build step in turn - maybe it can parse a text file with each command line you'll want to call. It can check the return codes from vb, or vc++ to check if it should continue or not. Installer programs like Installshield can be called from the command line so that step can be included too. If you can call your source control from the command line then you're sorted.

John C
Wednesday, November 14, 2001

Matt: the URL in my reply appears clipped in my browser window (although it still works OK when clicked).  Where it ends with "command_lin" should be "command_line.asp".

John C: thanks for the tip on making VB from the command line.  We didn't know about this.  Is there any way to set the conditional compilation arguments (short of doing an edit on the project file, I mean)?

Chris Dunford
Wednesday, November 14, 2001

I used to use Visual SourceSafe (VSS) and DevStudio, both of which can be driven from the command-line.

We used to store our builds as well as our source in VSS. Building a project from VSS was predictable:

get source from VSS to local drive

checkout targets (builds) from VSS

delete targets (builds) on local drive (to force rebuild all)

checkout project's version resource

increment version resource on local drive

rebuild on local drive

checkin version resource

checkin builds

To implement this I wrote a program to parse a data file. This data file contained a separate line for each project. Each line defined the following per-project information:

VSS folder name[s] of source

VSS filename of *.dsp or *.dsw

VSS folder name[s] of target[s]

Name of 'Configuration' (e.g. "Win32 Release") in *.dsp

Name of version resource file (*.rc)

These fields could have wildcards, and were given default values if not specified (e.g. folder name of source is assumed same as folder name which contains *.dsw or *.dsp; version resource is assumed to be version.rc in source folder; targets are assumed to be ./Release/*.* (if compiler's intermediate files are written to some other directory)).

To run my executable, specify: the name of the project to build (or, "*" to build all projects in sequence); and the VSS checkout/checkin comment to be associated with this build.

I could have made it create a VSS label too.

I also wrote a little utility to automatically parse/increment/write a version resource file.

To handle "shared" include file information (typically stored in some directory other than this project's source directory), two options: do "VSS get" to get the (potentially updated) shared include directory contents to local drive for each build; or, instead, we used the VSS "shadow" feature to ensure that the latest copy of our "common include" VSS folder was permanently available/visible in a "VSS shadow" network directory.

Christopher Wells
Wednesday, November 14, 2001

It is rather... difficult to try and script using DOS Batch files, or even the NT extensions.  (For one thing, you can't capture standard error for many programs into a logfile!)  So my first suggestion for anyone trying to script is to install the Cygwin Unix-on-Windows environment from www.cygwin.com.  At this point, you will have the exact same shell scripting environment someone on Linux or old-fashioned Unix systems like AIX, SunOS, or HPUX has.  You can also get any of a large number of books on "How to write shell scripts", as well as a cornucopia of online help.

Other people in this thread have commented on how to compile your VB stuff from the command line, so I won't go into that.  Others have said commented that you can run VSS from the command line.  (Although I wouldn't recommend VSS for anything more complicated than 5 engineers and a single project, it's buggy.)

With a shell script, you can set up a nightly build.  Check the sources out (via VSS or CSS), compile your VB stuff in the right order, call regedit if necessary, shuffle files around, and perhaps even call InstallShield from the command line to make a single .exe that can be tested the next morning.  And call a simple SMTP mailer to send everyone the build results...

I'll also note that it is possible to use "nmake" instead of the Visual C gui, and to use "javac" instead of Visual Java.  But there are compatibility issues; you might need to twiddle the makefiles and impose some developer discipline.

Carl Hommel
Wednesday, November 14, 2001

Chris: You can either point your browser at:

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vbcon98/html/vbconusingcommandlineoptions.asp

or type 'vb6 /?' from the command line (in VB's folder) to get a list of all the parameters you can pass to VB.

John C
Wednesday, November 14, 2001

Thanks for all the ideas

We have made a start using Kinooks Visual Build - it seems to offer a useful amount of samples for VB and VC++

Matt Trinder
Wednesday, November 14, 2001

The full "Best Practice" is not "daily build" but
"Daily Build and Smoke Test".  You need your
official set of test data to run all the programs
after you build them, night after night after night.
This is pretty tricky to do with distributed software,
and we never got to it at the large-scale service
I helped write.  Just getting a nightly build
instituted was like pulling teeth.

Lance Norskog
Wednesday, November 14, 2001

We went through this discussion ourselves when setting up our project. We didn't have VB stuff, but we did have quite a few C++ dlls, plus some files that needed to be generated via a codegen tool.
<p>
Another factor was keeping the build in sync with the actual development. We didn't want to force the developers to use makefiles for everything (since VC++ works pretty well for managing an individual project), but we also didn't want to force everyone to have to build everything all the time.
<p>
What we ended up doing is creating an XML manifest of our project. The XML file contained a list of each of the projects, along with the tool to build them, and the email address of the responsible engineer in case the compile failed. Then we ran the XML through a codegen template which spit out a batch file which actually ran the build.
<p>
The nice thing about this is that we could run the <i>same</i> XML file through a different template, and get out a different batch file that the developers could use day-to-day. And another one that helped put together the installer. It worked out really well.
<p>
You can get more details <a href="http://www.develop.com/software/GenXdownloaddetail.asp?vWizId=210">here</a>. You'll need a copy of Gen&lt;X&gt; to actually use the project presented there. The important thing is the concept - represent your metadata (the list of projects to build) in some concrete form, then operate on it, rather than going directly to the build step. You don't even need to use batch files. For those who haven't looked lately, the Windows Scripting Host V5 and the current JScript and VBScript interpreters are more than powerful enough to do any kind of command line scripting.

Chris Tavares
Thursday, November 15, 2001

A Daily Build and Smoke Test is a good thing.

But frankly, I haven't thought of a single reason to stop there.  Once a build server is set up, have it run a full build and smoke test every time someone checks something in.  You can even go beyond that, and have it run more often, so that if some externality changes (like a database goes down) you will find out because the smoke test fails.

CPU cycles are free, you don't get a refund from Intel for letting a machine sit idle.

If it takes any manual effort at all to run a build, that's too much - continue automating until you don't care how often you build because it takes no effort at all.

Kyle Cordes
Friday, November 16, 2001

There's another factor besides effort that dictates how often you build: as project complexity increases, build times lengthen, even as you throw more hardware at them. At some point, you have to struggle just to do a nightly build.

Mike Gunderloy
Friday, November 16, 2001

We have a rather nice build system we call "Clean Build" which gets it's definitions from a relational database. If I do say so myself, it's by far the most sophisticated solution I've ever seen or heard of (except possibly an IBM product).

It has a nice Windows GUI client (that's right, not a web client :) for configuring what items can be built, what goes into each release, categorising things, a monitor for builds scheduled or in progress, and covers packaging as well.

We don't exactly do daily builds. More like hourly, when we're changing things. Developers only need to know how to click a few buttons - there's no XML manifests or batch files to run, and the builds happen remotely...

If we could get around to it, it seems like a marketable product... but then, I'm not sure how many companies would appreciate it. We came up with such a solution because of our need to develop potentially 30 variations of a product in parallel, with around 10 compilers.

This is our second iteration, the first one used DBase files and runs on OS/2 (cause that is what the product is on).

Aaron Lawrence
Friday, November 16, 2001

It's all about ANT:

http://jakarta.apache.org/ant/

It's free and VERY simple to use. I had an automated daily build running on a C++ (client)/Java (server) projectwithin a couple of hours. That included labelling VSS, getting latest, running the build, archiving the previous build, and full logging to investigate broken builds.

Did I mention how EASY to use it was? :o)

Paul Reynolds
Tuesday, November 20, 2001

CVS for version control ( www.cvshome.org )
ANT for the build process ( jakarta.apache.org/ant )
UNIX cron for the scheduler (could have been AT on Windows)
APACHE and a mail server for the reporting (or your favorite web server: IIS, Tomcat, ... )

It's perfect and I recommend it highly for *Java* development

considering some automated integrated CVS/ANT like CruiseControl, AntHill

Robert Chevallier
Wednesday, November 21, 2001

I used to use bat files and what a nightmare that was. When they worked, they worked great. But they were a bitch to build and worse to debug.

I use FinalBuilder now and cant believe I ever lived without it. I not only use it for my builds, but I dont think I've written a single batch file since I found FinalBuilder. Its really more of  a visual scripting tool, but it has native commands for building VB, Java, Delphi, etc projects as well as all the standard batch file stuff. It even does my telnet scripts and FTP uploads.

http://www.finalbuilder.com I think, if not Google should find it quickly..

Chad Z. Hower
Saturday, November 24, 2001

Here we do it with a mix of
a) a tool to check out all the VB project files, Munge the version number and setup the copyright info.
b) .VBG files (usually with less than 30 projects each)
c) batch file to run VB on the VBG files
Typical build time is 1/2 hour on a fast machine with loads of memory 2 hours on a slow machine. We don't all do builds daily but it's still fairly frequent.

One important document we do have is:
"How to install and build everything on a blank machine"
(e.g. apply ALL the patches, install those handy third party OCXs etc)

Peter Ibbotson
Monday, November 26, 2001

Anyone tried using ANT to build VB?

davebo
Monday, June 24, 2002

Anthill is a build management server that ensures a controlled build process and promotes the sharing of knowledge within an organization. Verison 1.4.6 includes new adapters for Visual Source Safe, Perforce, PVCS, StarTeam, MKSIntegrety, and FileSystem as well as other feature updates. Tagging the source repository is now optional. The default project site navigation no longer gives an exception when browsing within directories more than three levels deep. The publish.xml project parameter is now optional. Ant parameters can now include quotes. Revision logs now contain all the revisions since the last build as well as comments

http://www.urbancode.com/projects/anthill/default.jsp

Rudy Manarek
Thursday, September 26, 2002

Another daily build tool worth checking is the imaginatively named Daily Build.
http://www.positive-g.com/dailybuild/index.html

Martin
Thursday, October 17, 2002

An impediment to daily builds in many teams is the time that they take to run.  Two open-source tools, ccache and distcc can dramatically cut the time it takes to build a tree from scratch, removing an excuse to daily builds and also making regular development much more pleasant.

  http://ccache.samba.org/
  http://distcc.samba.org/

martin pool
Wednesday, March 12, 2003

Over the past couple years I led a team that built a series of web-based application that use Flash for the client-side (e.g. http://www.mycardmaker.com/) - we never did find a way to do an automated build of the Flash stuff.

Meanwhile, I was very jealous of the setup the Mozilla people have with their Tinderbox tool (http://tinderbox.mozilla.org/showbuilds.cgi?tree=SeaMonkey) which does auotmated builds and shows on the web site the current status of all builds and who (if anyone) to "blame" for any breakage.

Matisse Enzer
Tuesday, March 23, 2004

How to add references and binary compatibility for a VB6 project in Makefile script?

Priya Pandian
Monday, June 21, 2004

Also it is worthing  look at http://sourceforge.net/projects/luntbuild/, it is a open source project concentrate on software build automation and build artifacts management based on apache ant.

alvin shen
Sunday, July 04, 2004

NAnt is an extremely powerful and flexible tool for automating a build. My team used it in the construction of a huge on-line application in the banking industry. The team size grew to approximately 35 'coding' engineers at our location in Portland. There were two additional teams involved, one on the East Cost and the other was located in India. By using one input parameter, we were able to use one NAnt build file to build the code on both the workstation (local build) and the build box. One build script to rule them all.

NAnt also has the lovely capacity to quickly integrate NUnit tests into the build file, making TDD a quick reality. In addition, NDoc is a really useful tool that also can easily be integrated to create you MSDN-like help from the build.

-=smith=-

Michael A. Smith
Wednesday, July 14, 2004

*  Recent Topics

*  Fog Creek Home