The Old Joel on Software Forum: Part 3 (of 5)

"Compiling" scripts

My main work is designing and building desktop applications for my companies platform, however I do a fair amount with scripting languages when we need to customize something for a client, or to extend an app.

I love scripting because it allows me to quickly take existing logic and apply it to a new problem. Over time, I've built up a library of procedures that I use all over the place (dts packages, web pages, etc.), but I'm drowning in un-organized scripting files.

The problem is this:

I have a function JSDateFormat() that lives in a file d:\source\common\js\date.js under source control.

I use JSDateFormat() in a few web apps, a dts package, inside files for scripting my own apps, etc. However, the JSDateFormat() function definition has to get copied all over because not all web apps/dts packages can get to d:\source\... (they are in source control too, but have different directory structures.)

I would really like to keep and edit the master file that lives in the "common" folder in source control, and have changes cascade down to the apps that use them, when those apps are "compiled".

I'm not aware of a tool that helps do this. I'm thinking that what I'd like is a browser that let's me view/edit all of the files under "d:\source\common\js\", and then let me compile selected procedures into a file called "common.js" that I can then place in a given application's path.

This doesn't sound terribly complicated, but before I dive in and create an application to do it, does anyone know of something? (My own app will not have an editor, it will launch an external one, but my app will do the compiling part...)

Thanks.

Wayne
Monday, December 29, 2003

I use a few CVS tricks to keep common code (also scripting-language code) in each projects directory tree.

So for example, I might have two projects with a directoroes trees like this:

/someproject/
/someproject/main
/someproject/foo
/someproject/common

/otherproject/
/otherproject/main
/otherproject/bar
/otherproject/common

I have each in a different module in CVS but the /common/ directory (and everything in it) is stored as a separate CVS module. Therefore both /common/ directories are kept in sync by CVS.

If you want more details about this setup, please email me.

Almost Anonymous
Tuesday, December 30, 2003

Thank you for your CVS trick, I will have to see how to do that in Perforce, it sounds useful.

Part of the problem which i did not mention is that sometimes I have modules with many functions defined, but a given app/page might use only one or two.

Also, it's a pain to maintain the list of scripting references right in the page/script task/etc.

What I really want to do is to compile one "Common" file for Javascript, VB Script, or CSS. The common file would only include functions that I selected to be included and would consist of functions from possibly many different files. The program might go even further and detect dependancies and allow you to include only certain named constants, etc.

This would also alleviate the browser having to download seperate files for web apps. However, these files could be used for more than just web applications (and more than just scripting files too, a sort of "Text Compiler").

Wayne
Tuesday, December 30, 2003

A few days ago I saw an article on MSDN about how to build COM objects using scripts. You can wrap your functions into COM objects, register them in the system and reuse. A little bit crazy idea ... but why not :)

Michael Popov
Tuesday, December 30, 2003

Take a look at literate programming and some of the associated tools:

http://www.literateprogramming.com/

I'm really surprised more effort isn't put into molding literate programming into something that is used more often.

Anonymous
Tuesday, December 30, 2003

Another way to do this is to have a 'build environment'.

So you have
/projects
. / myproject
. / otherproject
. / common

date.js lives under common, all the code for your projects is in the appropriate directory. then you have a build script which copies all the necessary files to an 'output' directory for each project, pulling in files from both common and the projects own directory.

of course, versioning may become problematic if scripts don't have perfect backwards compatibility, but that can be managed.

mb
Tuesday, December 30, 2003

"Part of the problem which i did not mention is that sometimes I have modules with many functions defined, but a given app/page might use only one or two."

You can allevate this problem by being more granular about what functions you put in what files. Rather than have huge files full of functions, split your functions into logical modules with one module per file.

For a particular project, always have all the files (say in a /common/ directory) but only include those files in your project that you are using.

In my main scripting language I have a function called import() than I defined myself. This function serves two purposes; it allows for path-indepent file importing and prevents multiple inclusion. So my scripts contain code like this:

import('Net/Mail');

To import a Mail module. That module might then include other files:

import('Base/Object');
import('Net/SMTP');

But only the minimum number of files are included that are needed for that particular functionality. I have a very large library of functions!

Almost Anonymous
Tuesday, December 30, 2003

Thanks AA. Unfortunately, Javascript and VB Script have now way to import other files when running in most environments (except web pages).

Similarly though, I could define some kind of pre-processing instruction that my own program could use in the "linking" process.

I'm still thinking about it.

Wayne
Tuesday, December 30, 2003

Thanks mb. I was also thinking along those lines. My own compiler will consist of an application that edits a "project file", which holds the references that you outlined.

Then, when I compile using my app, it will copy the master to to the local directory. If I get fancy, I can do the linking bit as well.

Wayne
Tuesday, December 30, 2003

Have any of you actually read about literate programming? It can be used to do pretty much exactly what you are describing.

Anonymous
Tuesday, December 30, 2003

" Unfortunately, Javascript and VB Script have now way to import other files when running in most environments"

If not in a web environment, what environment are you running them in?

Almost Anonymous
Tuesday, December 30, 2003

I've often used the VBScript ExecuteGlobal statement as sort of a dynamic include when needed.

'------------------------
'File: Math.vbs
'------------------------

Public Function Min(ByVal first, ByVal second)

    If first <= second Then
        Min = first
    Else
        Min = second
    End If

End Function

'------------------------
'File: Test.vbs
'------------------------

Private Sub Include(ByVal filename)

    Dim fso
    Set fso = CreateObject("Scripting.FileSystemObject")

    Dim inStream
    Set inStream = fso.OpenTextFile(filename)

    Dim source
    source = inStream.ReadAll()

    ExecuteGlobal source

End Sub

Call Include("Math.vbs")

Call MsgBox(Min(7, 9))

GuyIncognito
Tuesday, December 30, 2003

But if you don't need "dynamic" includes and your using the latest version of WSH then you can do something like this:

'----------------------------
' File: Test.wsf (don't include this header in the file)
'----------------------------
<?XML version="1.0" ?>
<job>
    <script language="VBScript" src="Math.vbs" />
    <script language="VBScript">

        Call MsgBox(Min(7, 9))

    </script>
</job>

Use the "Math.vbs" file from the above example.

GuyIncognito
Tuesday, December 30, 2003

GuyIncognito,

One suggestion: Add a global array to detect and prevent multiple inclusions. Then different source modules can import the functions they need without any collisions.

Almost Anonymous
Tuesday, December 30, 2003

Anonymous:
How does literate programming help? From my limited reading, it has the concept of modules, but still the 'master' app needs to get the modules from the environment somehow.
The 'compiliation' step is really building a single environment where the interpreter can pull modules from, be they 'literate' or normal.

(BTW, for those trying to do this, nant is a pretty cool tool which may be able to help you a lot. as would other build tools.)

mb
Wednesday, December 31, 2003