Fog Creek Software
Discussion Board




Code that writes code

I'm curious about how many people here have written utilites that write mundane/repetitive code in the language of choice. I'm currently writting a utility to build client and server side objects from a database table.

I wrote a utility at another company that would create text parsers (into SQL Server tables) from flat file definitions that we got from customers. Everybody else was writting these by hand, and took about a day to write and test them.  I got tired of doing this by hand, and this utility wrote the parsers in about a minute.

I imagine that people that have been programming for 10+ years probably have lots of these programs and libraries.

- Hector

Hector
Wednesday, May 07, 2003

I wrote a small application last winter that would write a class library from an xsd. Saved me a TON of typing...

Philo

Philo
Wednesday, May 07, 2003

This is an example of why laziness is sometimes a positive attribute of programmers.  It amazes me that so many programmers will waste so much time on mundane tasks when they could easily write a small utility to automate the job. 

SomeBody
Wednesday, May 07, 2003

lazyness?
or (programmer and/or his/her boss') lack of intelligence?

Ross
Wednesday, May 07, 2003

Not a lack of intelligence - it is intelligent laziness.

Because one is too 'lazy' to write thousands of lines of repetitive code, one writes a utility to generate the code automatically.

T. Norman
Wednesday, May 07, 2003

Well, our standard LISP joke is that the code would look like this:


((()) )()( )((( ))))) ()( )( )( )()))(((( )() ();

Absolutely no content to that post whatsoever, but it may be good for a chuckle.

www.marktaw.com
Wednesday, May 07, 2003

I guess the ";" is not correct.
So.... you hate LISP? Why not a joke about VB or C or C+++ or Java or Perl or anything else?

Ross
Wednesday, May 07, 2003

good info on code gen utils:
http://www.codegeneration.net

I have a bunch of small utils but my favorite is a Collection generator for C# called CodeSmith. Basically it will have to suffice until generics are added to C# in the next release. One of my other favs is one I wrote to map database and http form fields to class fields and output code. And I can't forget a ton of build utils I use, mainly to set up new projects and their builds using a template.

Damn I am lazy.

Ian Stallings
Wednesday, May 07, 2003

Just after the step where a new product idea has stopped being the answer to world poverty and dirty windscreens and as I begin to model the thing and create the small enfeebled shadow of the original idea and grow it, feed it and create new utility playmates for it; just after all that I think what I really should be doing is writing utility software.

Simon Lucy
Wednesday, May 07, 2003

'Programs that wite programs
are the happiest programs in the world'

Jon Udell often quotes this.

fool for python
Wednesday, May 07, 2003

I've written an Oracle PL/SQL stored procedure that takes stuff from tables and writes a UNIX shell script, which it then calls. The script does simple stuff like ftp-ing of files.

Patrik
Wednesday, May 07, 2003

This is one of the recommendations from "The Pragmatic Programmer": write code that writes code.

Basically, any time you have code that can be derived from metadata (a database schema, xsd, grammar description) you should write a tool that generates the code based on the metadata. Your overall maintenance burden will be lower, because you can just change the metadata as needed and regenerate the code.

Chris Tavares
Wednesday, May 07, 2003

But has anyone here written code that writes code to write code?

Alyosha`
Wednesday, May 07, 2003

> But has anyone here written code that writes code to write code?

Lisp programmers do this all the time.

Mac
Wednesday, May 07, 2003

Here's another example. I used (or should I say: abused?) the C preprocessor and Boost's preprocessor library to generate a tuple type of my own. Seems to me that "generative programming" has some future, be it with old tools like the pp or with fancy template metaprogramming ...

- Roland
Thursday, May 08, 2003

Aloysha, ASP developers do that all the time - VBScript to generate Javascript that will create HTML in the browser. The context shifting used to drive me NUTS.

Write a tool that generates ASP and you're really in deep...

Philo

Philo
Thursday, May 08, 2003

Perl scripts that generate triggers, PL/SQL stored procedures, packages.. Install scripts in SQL that generated more SQL to create tables (based on preferences), spooled the output and then ran it..

I like Perl for this sort of job, can you tell ? :) But any language that can handle strings reasonably well is a good fit for this sort of work.

The craziest thing I ever did was to generate Perl code and then autoload it and run the code I just generated (it was a sort of parser plugin thing).. That didnt work too well, but it was good for laughs (and also made me understand autoloading far better than I had before)

perlnut
Thursday, May 08, 2003

"But has anyone here written code that writes code to write code?"

Does PHP that writes XSL that transforms XML to HTML count?

Just me (Sir to you)
Thursday, May 08, 2003

the only time I've ever done this was for a tool to generate the html forms & code to add/update/delete SQL Server fields (to update news / content on a website for example).  It would look at the database structure, analyse the fields and build appropriate form field types (textbox, checkbox, etc).  Saved quite a bit of repetitive coding, helped prevent errors creeping into SQL statements, and also taught me a fair bit about what SQL Server system tables contain... so although it was initially a pain to build it was worthwhile.

Worst case I've seen was a full wysiwyg webpage builder written with ASP - that used ASP to write Javascript to write ASP or some such nightmare...

surreal
Thursday, May 08, 2003

I have written a VM.  Only what I did was write a bunch of lookup tables for the opcodes and the registers they affected etc (hey, this was a very strange opcode set alright!) and then wrote an app that turned this vm into longhand.  The resulting 3mb sourcefile was all code paths listed explicitly.  This compiled to about 2.5mb.  Then I had only to turn each opcode in the binary into a computed goto, and then run the program on this monster VM.  The codepaths that any particular app used quickly became cached, and ran like the clappers.

Nice
Thursday, May 08, 2003

I got sick of string concatenation, so one day...
http://www.jostraca.org

Richard Rodger
Thursday, May 08, 2003

Sure. I build websites making a config file that builds and combines modules to full fledged programs. All in perl, of course  :)

Ros
Thursday, May 08, 2003


Code generation programs are really braindead.  The real challenge is to design your application so that code generation can be productive in your application.

Joe AA
Thursday, May 08, 2003

Chris Tavares: The PP's are right but only up to a point.

We have a ruby application that takes an xml file detailing telecoms message flows, and generates code (C++) for our network simulator. It is easy to use and generates a complete stub, but then you have to write the contents of the stub yourself (e.g. what to do when the message arrives)

The problem comes when you have to update the flows: If you regenerate everything useing the code generator, you need a really good diff and merge utility to be able to put the code back into the regenerated stubs without too much pain.

best regards, treefrog

treefrog
Thursday, May 08, 2003

treefrog, check out the pimpl pattern.

you could make your stubs simple one liners too.


Thursday, May 08, 2003

treefrog - To "solve" the mixing of hand written and generated code problem what I've done in the past (using Java) is to only ever extend the generated code via subclassing. This means that so long as the methods you're overriding retain the same signatures across regens, which in your example it sounds like they would (something like processMessage(...)), you never need worry about merging with or changing generated code. Works great for Swing UIs, O/R mapping frameworks and lots of other code best written in a "language" other than Java.

Alex Moffat
Thursday, May 08, 2003

The whole subclassing stuff has really really got to me.  It is a common approach; I think I've seen it in RAD tools for Gnome and/or KDE too?

Things like VB and Delphi have separate form files that define the layout, IIRC.  These aren't intended to be human editable.  On the other hand, many languages like Java require you to build your forms dynamically.  And I have found myself doing dynamic building of forms in other languages too, including VB and Delphi.  But RAD tools don't try to touch this code.

If a compiler can understand code, the same kind of parser could actually grok your code and do in-code RAD instead.  It is more complex but, I would be worth it I think!

I did start something like this: http://www.geocities.com/varfar/ctree/index.html ; but I run out of steam when I started to think about using bison or something instead.  I had solved the interesting part- the parsing ;-)

Nice
Thursday, May 08, 2003

Chris Tavares:
The author of the Pragmatic Programmer was interviewed about this very subject: http://www.codegeneration.net/tiki-read_article.php?articleId=9

Interesting read.

Ian Stallings
Thursday, May 08, 2003

John Lam writes a lot about code generation (code writing code) on his software blog:

http://www.iunknown.com

runtime
Thursday, May 08, 2003

Treefrog:

Your code generator is broken. You shouldn't have to edit generated code at all. You can fix this one of two ways:

Use subclassing to hook your customizations into the generated code. I've seen this written up at the "Generation Gap" pattern.

Put your code into the metadata. This is the approach lex and yacc take. This way, you have a single source for the metadata and the customizations.

Personally, I prefer the subclassing approach.

Chris Tavares
Thursday, May 08, 2003

Great article by Dave Thomas.  Regarding code maintenance, he says "Favor active over passive generators, and make sure that the outputs of the generator never get stored in the source repository."

So one solution is to put the stub contents in along with the meta source (e.g. your xml file).

P.S.  Clever name treefrog! #;-)

P.F.
Thursday, May 08, 2003

Generation gap pattern is described here http://www.research.ibm.com/designpatterns/pubs/gg.html

Alex Moffat
Thursday, May 08, 2003

"Favor active over passive generators, and make sure that the outputs of the generator never get stored in the source repository."

Or even better, use the tools the language gives you. In C++, use Templates for code generation. Take a look at Spirit for a nice and powerful example.

Sebastian Wagner
Friday, May 09, 2003

I don't know if anyone here has seen it yet, but our product [1] has a solution for mixing hand-written code with generated code.  We created a concept called UserCode, it basically allows you to insert hand written code in the generated code.  The hand written code is parsed out and reinserted in the output when you re-run the generator.


[1] Codify
http://www.workstate.com/en/codify

Peter Schroeder
Friday, May 09, 2003

I helped with an IDL compiler, which generated C++ and Java client/server stubs from an arbitrary interface written in an IDL.

Christopher Wells
Friday, May 09, 2003

I wrote my first code-generation code back in '95, and I've written new ones every time I changed jobs since. My first ones were basically search-and-replace tables, then I added conditionals, loops, and binding to database sources.

I was a big fan of WebGecko's apgen (http://www.webgecko.com/) for quite awhile, but ASP.NET output caching is much more useful now. I looked into M4 and GSLgen lately, but I prefer to use the same languages for writing code and writing generator code, so right now I'm starting to get my feet wet with CodeSmith (http://www.ericjsmith.net/). I like having full access to the .NET languages and runtime, and the ASP.NET-like syntax is familiar territory. It's not open-source yet, but is free and I can call it transparently from my custom VB.NET compiler.

I've found that code-generation is most useful for three tasks:

1. Po-boy workaround for not having generics.
2. The work to get data from form POST to custom objects to stored procedures to tables to stored procedures to custom objects to HTML forms (80% of many web apps).
3. Dynamic execution, a la Javascript eval().

Richard Tallent
Friday, May 09, 2003

One of my proudest work-related moments occurred a couple of years ago, involving a code generator.

I began working on the server-side team of a ongoing bank-related Java project, where all database access was implemented thru accessors to Cobol transactions on a mainframe. Inserting and extracting resultset data from the transactions required quite advanced data mangling (you know, setting and getting data from fixed size positions in vectors and matrixes).  Everyone on the team used copy&paste programming for generating the accessor classes for each transaction, which was boring, time-consuming and error-prone. So after writing my first accessor, I came up with the idea of writing a code generator for it, which I did in a couple of days. I made a simple XML DTD for declaring the format of all in and out-data and logical names for each field, and wrote a code generator that generated both accessors and resultset iterator classes for the transactions. I am specifically proud of the resultset iterators, as the use of them significantly improved the quality of the code, where everyone previously had matrix parsing code all over the place.
I'll never forget the faces of my colleagues when they 'got' the idea, specially as most of them had worked in the project for about a year without thinking of a practical solution to the evident uglyness themselves.

I know, bragging about things is a bad thing, but sometimes you are entitled to :-)

Yablan
Friday, May 09, 2003

I'm really excited about the possibilities of run time code generation. This is something that wasn't practical to do in C++, but .NET makes easy.

At my job, we've used a code generator to create custom parsers for different file formats. This worked well, but still required us to re-gen and build a new parser every time a file's format was changed. Since the code generation occured at our office, each newly generated parser would have to be shipped to the customer.

Now, with .NET, the customer does the code generation without even being aware of it. At startup our parser is able to read a format definition file and use it to generate a custom parser assembly in memory. This gives the customer a highly configurable parser that is as efficient as a hard coded one.

John Bush
Saturday, May 10, 2003

I use my jpeg encoder to generate code for my jpeg decoder...

The jpeg stream is a program to be executed by the jpeg decoder.

I like this way of thinking.  :)

idle
Friday, May 16, 2003

Ha ! I made a Java program that parses an HTML imagemap to write code in PHP /GD which will recreate that imagemap (in HTML)  on the fly, using data in a database that alters that image map on the fly. It 'snot just useful- it lets me do somthing that otherwise is impossible.

Sweet. I didn't know there were other people who were into this kind of programming.. I ahve been trying to tell people how useful it is...

-C

chad
Tuesday, July 08, 2003

Hi All,

Since March 2003, I generate 50-80% of my web applications. I am developed an XSLT based code generation utility, which extracts data model from Oracle  database and generates PL/SQL packages, sequences, Data Access Code, ASPX Pages and CodeBehind classes.

Also, I have Customized datagrid class and standardized web pages so that developing single-table, master-detail data-driven web pages/components are easily developed.

Currently, I am working on a newer version, which will generate code for windows, mobile and web  applications. Here--in Turkey-- I have contacted with a local IT Company, to market my product starting from Jan 2004.

I also plan to co-operate with a local university to further extend code generator.


Monday, October 27, 2003

I've just finished a project that generates a input to java_cup parser generator and JLex. (Java version of lex)

java_cup generates a java parser and JLex generates a lexical analyzer.

My tool also generates code for a syntax tree which is generated by the resulting java_cup program.  The syntax tree contains code for source-to-source transformations.

The result is a source-to-source compiler so what I've actually written is a compiler-compiler-compiler.

The tool is bootstrapped. IE it is written in it's own language.

So this is code that writes code that writes code that....

D. Yessick
Sunday, November 16, 2003

I have a whole host of references to using code generation for scientific applications from the early 60s...  In particular, hop over to http://www.autodiff.org (and their reference list) for a sample.  The earliest reference I have seen is from 1957 if I remember well.

I remember 'fixing' a COBOL code generator (written in COBOL, naturally) to do checkpointing in the generated code -- the generated code had a tendency to crash, and since it took around 24 hours to run it, this made a big difference.  And this was circa 1986.

A different twist on 'code that writes code to help write code': Centaur, a structured code editor generator :-)

Oh, and there is a really nice program called PGG at http://www.informatik.uni-freiburg.de/proglang/software/pgg/ which is a 'program generator generator'.  Now *that* takes a while to wrap ones head around!

Jacques Carette
Thursday, April 01, 2004

Hi everybody !!!
I'm also a "lazy" programer that was interested in the code that write code.
I'm a Oracle pl/sql programer working for a company(the company activity is irrelevant).
An year ago what started more as a joke has developed in time in a "nasty" tool.
We have tried to create a module aplication for our own purpose in order to ease our work.
The tool(is too much to say right now) is creating tables, standard triggers, topics for specific collumns, standard packages for updating/deleting/inserting data into the tables, a repository for all objects(paralel to oracle data catalog).
What it needs is an interface and also I have a lot of plans for it.
What I want to create in the end is also a grafical interface like Visio from which the code and the Oracle objects will be modified and maintained(sounds utopic but it can be done).
I'm looking for opinions and ideea exchanges so if anyone is interested pls give me a reply.
 

ungureanu cristian
Tuesday, June 22, 2004

*  Recent Topics

*  Fog Creek Home