The Old Joel on Software Forum: Part 3 (of 5)

HTTP scripting?

I'd like to write a little script to automatically go out to my library website and pull down the list of books that I have checked out.

I started to do this using the HTTP classes in .NET. They are nice, definitely up to the job. However, the library's web application has two or three steps and going between those steps uses cookies and hidden inputs and other state-maintenance stuff, which will be a bother to mess with.

It occurs to me that this particular wheel may already be invented. Does anyone know of a HTTP scripting tool, class library, etc, that does a good job of simplifying the faking of a user logging in and navigating a web app?

Matt Conrad
Sunday, December 14, 2003

I automate Internet Explorer from perl to perform "black box testing" of web applications. The nice thing about automating IE is that it correctly handles Javascript. You should be able to automate it from any language that can call COM though.

Matthew Lock
Monday, December 15, 2003

what is the DLL you need to link to for ie ole automation?

The Artist Formerly Known as Prince
Monday, December 15, 2003

If I were trying to do this, I'd write a little VB app with an embedded Internet Explorer component and poke things at it. This has the advantage that you don't have to scrape or parse the HTML yourself; you can wait for the document to arrive and then find the things you want in the DOM, all nicely parsed in a tree format for you. Also IE will manage cookies for you.

A Unix-y person might consider using "expect", libcurl, or even lynx from the command line (there's a command line option that just returns the web page as plain text on stdout). It all depends on what your skills are.

Joel Spolsky
Monday, December 15, 2003

> what is the DLL you need to link to for ie ole automation?

I just create an object of type: InternetExplorer.Application

Then as Joel says I query the DOM once the page has loaded.

Matthew Lock
Monday, December 15, 2003

Try cURL. http://curl.haxx.se/

Nitin Bhide
Monday, December 15, 2003

if you are of the perl persuasion try *LWP* - http://www.perldoc.com/perl5.6/lib/LWP.html
examples ....
http://www.oreilly.com/catalog/perllwp/

python has *urllib* -
http://www.python.org/doc/2.3.2/lib/module-urllib2.html
examples ....
http://www.python.org/doc/2.3.2/lib/urllib2-examples.html

dont be fooled by these scripting languages they are cross platform and have all the tools to do the job.

peter renshaw
Monday, December 15, 2003

Automating IE sounds quick and easy, which is what I'm looking for here. Thanks for all the suggestions.

Matt Conrad
Monday, December 15, 2003

wget

christopher baus (www.baus.net)
Monday, December 15, 2003