Fog Creek Software
Discussion Board




Shell Scripting Fanatics

I manage a team of developers, and two of them, UNIX types, who swear by shell scripts (ksh, tc, etc.). They shun more formal programming languages (Java, Perl), frameworks and tools, claiming that these should only be used if the scripts reveal bottlenecks.

As a manager, I'm trying to get a handle on their zeal, and get a handle on the pros/cons of their approach. They're clashing with the rest of the team. Maybe it's an issue of programming style (bottom-up v. top-down), but at this point I'm seriously considering dumping them.

Can anyone point me to significant applications that have been written in shell scripts alone, i.e. not in a general purpose programming language?

Bananas Foster
Wednesday, July 21, 2004

I've done parts of an Oracle Application with UNIX shell scripts, but just parts ofcourse.

I use shell scripts when its more efficient with regards to development time to do so. For mundane tasks like parsing text files and such shell scripting is more efficient than C.

So these scripts uses awk, sed and friends to do their parsing and loads the data into a DB. However I would not use scripts for anything more advanced than this.

Used in the right places shell scripts are your friends and saves you time, compared to development in a more formal language like C or whatever.

Patrik
Wednesday, July 21, 2004

Shell scripts - so ugly. I cringe inwardly every time I see something like echo `expr $x + $y` - you mean you're running a separate binary just to add two numbers together? - but some people are set in their ways and love them as they embody that UNIX ethos of patching together lots of little programs to do a job. Lucky you having to keep the things maintainable, heh

Matt
Wednesday, July 21, 2004

What kinds of applications are they writing with shell scripting languages?  There are plenty of legitimate uses of shell scripts and accompanying command line tools.  Obviously they shouldn't be trying to write web browsers in shell scripts, but as long as the code is maintainable and works, does it matter a whole lot what language they use?

Ryan Phelps
Wednesday, July 21, 2004

Some years ago, I worked for a company and we had to build a fairly complex application for a client.

The guy who did this work was in the classic unix hacker mode.  He insisted on doing it in Shell scripts, using awk, text search and what have you. He did write some C to do a X Window user interface to his app, but X Windows interface just called shell scripts. In other words, he ran the script, piped it into a file, and then displayed the result (after parsing in a X Window list or whatever)

At that time, I only managed the Windows team working for the same (and other) clients. I knew this guy was heading for a disaster because the application had to deal with large volumes of data (so parsing text files, awking them, etc was slow) and the X Windows interface had to be interactive.

I tried to stop this guy doing it in shell scripts as I knew it was heading for disaster, but had no power to do it.

The guy actually got his app sort of working, quickly, in a demo-ish way, and based on that convinced non-technical management, he should proceed with his route (hey he's got the screens in a demo - must be nearly there).

The guy gets his demo running in days (to convince management), but then spends maybe 6 months trying to overcome the most egregious problems with his design. It's too slow. It's too buggy (the shell scripts had this unfortunate habit of breaking if the data in the text files contained certain characters). It's not interactive enough. etc.

These problems are all things that I predicted at the outset.

A year or so later, the client is unhappy with his app. It's still too slow. It's still too buggy. It's still not interactive enough. etc.

The unix hacker guy has left, so my team rewrite his app.

We threw it his app away, and rewrote the app using Oracle RDBMS to store the data instead of text files, and a new fully interactive X window interface.  It tooks us 1-2 months to get the whole thing working and installed.  Functions that took hours in his GUI, took seconds in hours.  The only thing we used shell scripts for was to import the old data into Oracle.

The moral is, I think, you can push the shell scripting idea too far.

The other moral is, know the limitations of shell scripts (or any technology) before investing too much into it.

Your mileage may vary depending on circumstances.

Anon
Wednesday, July 21, 2004

"Reliable and maintainable" -- there's the problem.

If they are the only two writing shell scripts, it sounds like only those two can maintain them.  So, the problem.

I would think a little (2 week) training in Perl would give them a 'higher' level language, which still supported lots of the shell-script 'tweaks' they've come to know and love.

I've been amazed at the resistance some 'super developer' people have to communicating what they've done, how they did it, or passing it on to others for maintenance.  They are only 'super' in their little niche area -- and their resistance to passing information on means their project dies when they leave.

AllanL5
Wednesday, July 21, 2004

Eh, 99.9% of the perl I have seen is very much in the shell scripting area. Shell scripts are great, they are easy and useful. Why the HELL would I write a secure backups system in java, for simple flat files, when I could use shell scripts, ssh keys, scp, cron, tar , bzip2 etc.....

Sorry, your greatly designed, gui based, java backup solution is just a plain waste of time for half the custom backup software you will write.

Shell scripts are great because of their simplicity, they can be made very easily maintainable and everybody can learn a bit at least.

fw
Wednesday, July 21, 2004


>Can anyone point me to significant applications that have been written in shell scripts alone, i.e. not in a general purpose programming language

All unix startup scripts are written as shell scripts.

At my shop we have all sorts of tests all written as shell scripts (i wouldn't do it like that). One of them creates a net setup on three computers, configures servers, here there, runs the tests and gathers statistis on the run.

Michael Moser
Wednesday, July 21, 2004

Much ado about nothing.  Shell scripts are fairly easy to understand if you have read them for more than a couple of minutes.  I agree with the idea that a UI should not be from a shell script unless it is to support one-off activities, (menus for ops, starting DBs, etc)

Perl versus shells: if you do not need it why bother?  While I can do both, sometimes its just a matter of six lines in ksh and I am done. 

Perl versus Java or a compiled language: You need some pretty heavy activity to warrant it. 

Perhaps the question should be what does it matter as long as no one is taking you down the "only I can do it path."  If you have two people who know it or a common structure (like shell scripts) you are hardly beholden to them.

***I would be curious what you decide and on what you base the decision.

Anonanonanon
Wednesday, July 21, 2004

You could always demand that their shell scripts be portable between Linux, Solaris, and HP-UX. That should shut them up.

Shell scripts have a reputation for being portable, but it's actually very hard. The actual shell may be the same on all the systems, but there isn't a script in the world that doesn't use external programs, and those vary wildly from Unix to Unix.

Chris Tavares
Wednesday, July 21, 2004

I'm not against shell scripting per se. I frequently write them myself to automate repetitive or sysadmin tasks.

The problem I see materializing is that they use shell scripts to write everything-- admin tools, utilities, CGI, glue apps, etc. I see dependencies starting to creep into our key enterprise systems-- all reliant on piles of little shell scripts.

Frankly I'd start to feel like we're making progress if they'd just standardize on one or two languages, a process and some docs. In our shop, shell scripting has made it too easy to sweep that all under the carpet.

Bananas Foster
Wednesday, July 21, 2004

Tell them that they're pigeonholing themselves, and damaging their future professional development by refusing to broaden their horizons.

Justin Johnson
Wednesday, July 21, 2004

People who are experts with only one tool are screwed when that tool becomes obsolete or goes out of fashion.

Justin Johnson
Wednesday, July 21, 2004

So... it sounds like the shell scripting isn't your problem them. It sounds like your problem is a lack of process.

Yet you want to fire the people that write the scripts?


Hmmm.... If I was your boss, I think I'd know where to look to fix the problem. Firing you would only come after the frank discussion about why you weren't able to get a handle on your development team.

linux admin
Wednesday, July 21, 2004

P.S.

There's a reason why the 30+ year history of *nix has favored the use of a lot of little scripts.

This reason is left as an exercise for the reader. Of course, as an IT manager, you should already know.

linux admin
Wednesday, July 21, 2004

He's not talking about firing them because they use shell scripts.  He's talking about firing them because they refuse to consider other tools, and they're causing friction with the rest of the team.  It's a legitimate managerial problem.

Justin Johnson
Wednesday, July 21, 2004

I think I will create all my client-side apps with DOS batch files. :-P

Wisea**
Wednesday, July 21, 2004

One good way to wean shell scripters into producing stuff in a formal language is Code Generation.

Get them to read The Pragmattic Programmer [1] and Code Generation in Action [2].

They can then write shell scripts (which they like) to generate the Java.

As herrington points out, there is an overhead in the process to Code Generation.  It does, however, get over all that typing overhead that puts scripters of formal languages.

[1] http://www.pragmaticprogrammer.com/

[2] http://www.manning.com/herrington/

Ged Byrne
Wednesday, July 21, 2004

The power of Unix lies in the command-line, which is not equivalent to scripting.
It won't be easy to convince these people, as long as they are not bitten by the consequences.

GP
Wednesday, July 21, 2004

If you write more than 50 lines in a shell script you should be doing it another way.  For me that other way is now python.

christopher (baus.net)
Wednesday, July 21, 2004

I can see why they'd be averse to things like C++/Java {because suddenly you need a build environment and makes and everything like that }, but not upgrading to using Perl when you need to is nuts.

It's like shell, only more so. And has a built-in "sed". And since this is UNIX and pipes are pipes, you just start writing some of the stuff in Perl and keep the trivial in shell.


There's nothing at all wrong with writing massive applications in shell and tweaking the slow parts into C++. It's how things like USENET used to run, and they managed just fine.

Katie Lucas
Thursday, July 22, 2004

I do a lot of batch stuff in shell scripts, usually centered around:

1. Operating on a BD based on a file. I usually validate as much as I can (using awk, sed, etc), then I load to DB and call a sproc, that does the rest of the work.

2. Data-crunching and e-mail results - e.g., daily/weekly/monthly statistics. Again, it's a mix of shell script and sprocs.

It could be done in perl or python, but I don't admin the servers, so I prefer to stick with what's already installed. When perl or python becomes standard on these systems, I'll probably give python a try, as I like it better than perl.

Besides, ever since I found out about the "set" command, I've developed a certain fondness for shell scripts. It allows you to create an execution trace, displaying, for each line, the actual command line executed, with the value contained by each used variable. So, if anything goes wrong, you just open that trace and follow your script's execution, line by line. I haven't seen that in any other language, yet.

I know of no significant app (e.g., acct package) done in shell script, and, IMHO, it would probably be a bad choice of "language" for such a job.

OTOH, there are several GUIs out there that are front ends to CLI tools (such as wget), and do a very good job at it.

Anyway, as someone pointed out, your problem is not shell script, but the classic hammer/nail problem.

A final note: You're the manager, and you have an idea of where you want to head. You don't have to prove them that a "shell-script-only" approach is a bad idea; it's the opposite - they have to prove you that "shell-script-only" fits the bill, IMHO.

Paulo Caetano
Thursday, July 22, 2004

I *strongly* advise you read the following article:

http://www.tcl.tk/doc/scripting.html

Hopefully, you will then appreciate why scripting is so damn useful, and so much more productive.

David B. Wildgoose
Thursday, July 22, 2004

"There's a reason why the 30+ year history of *nix has favored the use of a lot of little scripts.

This reason is left as an exercise for the reader. Of course, as an IT manager, you should already know."

Pompus Linux asshat.

gentoo
Thursday, July 22, 2004

If they're writing things like CGI in shell scripts, you're headed for major problems, or they're already there. YOU ARE GIVING ANONYMOUS USERS ACCESS TO A SHELL ON YOUR WEB SERVER.  THIS IS BAD!!!

If there was a way to make the text larger, I would.  It's a really bad idea to give web users access to a shell.  If your visitor pokes hard enough, it's highly likely that they can inject their own commands into the input stream. Something like:

echo "Your site has been 0wn3d dUd3!" > /var/www/html/index.html

An that's only if they're feeling generous that day.

Clay Dowling
Thursday, July 22, 2004

Yes, they have written quickie CGI scripts. There are over 100 of them.

Thanks for everyone's thoughts. I'll offer to send them to a Perl course (that's what most of us use in my dept.) and have them put it to work when they return. 

Hopefully this will help change some habits. One of them, despite a prickly character, is a fine sysadmin who I'd like to keep around if possible.

Bananas Foster
Thursday, July 22, 2004

> but there isn't a script in the world that doesn't use
> external programs, and those vary wildly from Unix to
> Unix.

And a Java program doesn't use those external programs?

Ignore my ignorance
Friday, July 23, 2004

*  Recent Topics

*  Fog Creek Home