Fog Creek Software
Discussion Board




Dynamic Languages and Enterprise Applications

Dynamic languages, especially Python and Ruby, are getting lots of press lately.

For example, a recent article by Robert C. Martin ( http://www.artima.com/weblogs/viewpost.jsp?thread=4639 ) suggested that a dynamic language in combination with a test suite is

1) just as reliable (the tests catch "compilation errors")
2) quicker to develop
3) shorter, hence more maintainable

However, I haven't seen a lot of enthusiasm in the Fortune 5000 for dumping compiled languages like Java, C# and C++.

Some reasons for this are that the languages are not backed by major vendors and there is as yet not much tool support -- but I also get the sense that the companies don't think that this is "a difference that makes a difference".

What do you think? Are REE (Ruby Enterprise Edition) and uh, the unfortunately-acronymed Python Enterprise Edition coming soon to a nearby enterprise application?

Peter Breton
Wednesday, August 20, 2003

This is not a new concept - McKinney espoused it in Chapter 4 of "Hardcore Visual Basic" - "The Ultimate Data Type" wherein he announced we should all use nothing but variants.

He was immediately dogpiled by a LOT of people.

You lose a lot of efficiency by not typing your data, and IMHO you introduce a passel o' potential bugs.

Philo

Philo
Wednesday, August 20, 2003

I think you're over-simplifying the issue: using a dynamic language and using variants are not the same thing. Both Ruby and Python are modern OO languages, with classes, inheritance, exceptions, etc.

One of the main issues with static languages is that the explicit typing is a lot of overhead, which results, in the long run, in MORE CODE.  The extra code has historically been justified by the safety that compilation gives you; Martin is suggesting that a good test suite offsets this. The cited article and the comments beat this topic to death, so I won't elaborate further.

So there are 2 separate issues here:

1) Are dynamic languages more productive than statically typed ones? 

Of course the jury is still out on this one, but I think if I can deliver equivalent functionality and equivalent stability in the same time with less code, then that is a strong reason to think so.

Less code also has long term cumulative effects because there's less to maintain.

2) If 1) is true, why aren't large corporations interested?

(Of course you are free to think that they are not interested because 1) is false)

Pragmatically, I program "where the money is", and thus I haven't programmed professionally in Python or Ruby simply because I haven't seen much demand for it. The subtext of this topic is, why is that?

Peter Breton
Wednesday, August 20, 2003

Philo, take a look at Python + Psyco:

Psyco is a "specializing compiler" for Python. It compiles Python code specialized for the types at hand (and possibly specific values at hand), during runtime.

It's like expanding templates and compiling to native code at run-time. While it doesn't yet make Python faster than C, it brings them much closer together, and with zero additional effort.

And Python programs are usually much easier to write.

Also, the claim that a dynamic interpreted language is slower is usually true, but not always.

The K language, for example, has the amazing idiom '{x y z x}/', which - depending on the arguments, is either a sort that takes multiple fields into account (sql "order by"), or a filter that intersects conditions on all fields, (sql "select where"). There are two things to note about it:
1. It is not a special case or anything - it derives from the language semantics, which are simple and orthogonal.
2. It runs in speeds comparable and usually exceeding a C implementation of the same process.

Ori Berger
Wednesday, August 20, 2003

Peter, help me out here - you're saying that test suites absolve the issue of data typing. I *guessing* that that means that a thorough test will point out places where dynamic typing got you in trouble?

But then what? Don't you have to *then* write the "more code"? If the test suite points out an error where you're assuming a date but the user entered an integer, how else do you deal with the error but by writing more code?

With strong data typing, the only place you have to validate a data type is at the interface (often simply via declaration). With dynamic typing, don't you have to validate it at *every* interface?

I've worked in a dynamically typed language (VBScript), and I've worked in strongly typed languages. I know I prefere working strongly-typed, and find the code more efficient.

Philo

Philo
Wednesday, August 20, 2003

You don't need to write more code because you write the same tests you would write for a typed language anyway.  If I am testing that a method returns 23.4, the test will prove it is returning 23.4 which is in fact a double.

With the C++ or java test, I already know it is a double, but I still need to write the same code to show that the value is 23.4

Oren Miller
Wednesday, August 20, 2003

Oh, and with a dynamic language you don't write any extra validation code.  The interpreter will validate the type for you at run-time (and since the tests excercise this code, this is just fine and dandy).

In fact, languages like ruby are great since they call methods by passing messages about.  So I can pass in any object that has the same interface, in the true sense.

If a method takes in an object and calls its toString method (or whatever), it doesn't matter what object I pass in as long as it has a toString method on it.  I don't need to worry about what it inherits from or any of that.  This is why there is absolutely no need for things like templates in languages like ruby since the equivalent functionality is built in to all methods.  This makes writing generic code a breeze with no need for special syntax and long compile times.

Oren Miller
Wednesday, August 20, 2003

Philo,

You can think of it as a tradeoff: the type declarations add overhead to the code, the dynamic ones need validation. The point the article is making is that you always pay for the type declarations even when you don't need them, and that feels like a Premature Optimization.

To answer your question specifically, you can just validate the interfaces that you care about (eg, the public ones).  The overall claim is that

  dynamic typing + testing < static typing

You should play with Ruby or Python or Perl 5 to get more of a feel for the tradeoff. VBScript, Javascript, etc are in the category of lightweight scripting languages. They have their place, but no one is suggesting that you build enterprise applications with them.

For many of these applications, performance of the language is a red herring; the performance of the application tends to be overwhelmingly dominated by network and/or database calls, and in-memory calls are simply lost in the noise. So there is, in general, no performance-related issue with using Python, Ruby et al.

The real concerns tend to be maintainability and ease of change.

Peter Breton
Wednesday, August 20, 2003

I was thinking some more about this.  And it occured to me that with a static typed language you are writing considerably more validation code than a dynamic one.

C++

double myMethod( std::string variable )

Ruby

def myMethod( variable  )

the double and std::string *is* type validation code.  This is validation code that is constantly being written everywhere throughout the application, and is always required.  So I'm not sure I agree that dynamic languages require more validation code at all.

Oren Miller
Wednesday, August 20, 2003

Philo, dynamic typing is not the opposite of strong typing, but of static typing. Python, Ruby (and also Smalltalk) are all strongly, dinamically typed languages.

Giovanni Corriga
Wednesday, August 20, 2003

Re: Python Enterprise Edition

Heh,  actually, it's Python Enterprise Application Kit:

http://peak.telecommunity.com/

The project is still in its infancy, though, and only suitable for experimenters (and the authors) at this stage.  Still, it has workable, easier-to-use subsitutes for many J2EE technologies, such as JNDI and EJB (minus distribution; IMO distributing EJB's is generally a waste of bandwidth and computing power).

I actually do use PEAK in my "day job", in a medium-sized "enterprise".  (Around 2K employees worldwide, 24/7 apps.)

Phillip J. Eby
Wednesday, August 20, 2003

Oh yeah...  VBScript is not a dynamically typed language, it's a weakly typed language.  Big difference.  Perl is also weakly typed -- as is C, ironically enough! C is weird in that it's a statically typed language with weak typing.

Strong = values are always of a given type, and require conversion (NOT casting) to obtain a value of another type
Weak  = values can be interpreted differently based on context (e.g. casting, or use in an expression with a value of another type.)
Static = variables can contain only one type
Dynamic = variables can contain any type

Anyway, Python is a strongly typed language.  Doing this:

"2"+3

Results in a TypeError in Python, even though it's legal in a host of weakly typed languages, such as Perl and C.  (Although, the outcomes in Perl and C for that expression are *quite* different!)

Phillip J. Eby
Wednesday, August 20, 2003

re Python and Psyco:

there is a new Python project called "PyPy". They are implementing a Python-compatible runtime in Python itself, with the hope that technologies like Psyco will allow reasonable performance.


http://codespeak.net/pypy/index.cgi?home

runtime
Wednesday, August 20, 2003

I work in an EAI group for a large telco. For us "enterpise app" means major mainframe based billing apps, SAP HR stuff, Siebel CRM, exchange activation systems - things in the $100 million range. My jaundiced view is:

1) increasing programmer productivity by a factor of 2 would have no impact on "solution delivery". Enterprise and EAI apps are not technologically driven.  We have a 3-to-1 ratio for definition phases versus build phases - that doesn't include deployment phases. Build is maybe 1/6th of the total and  the model is so fragmented that programmers are limited by very simple steps on the Joel Test. Programming language characteristics are the least of their worries. Getting guys login ids inside of three weeks would have more impact than any programming language change.

2) If the CEO had a magic wand, he would just squeeze programming language choice out of the universe like a watermelon seed between thumb and fingers. These guys are just sick of the proliferation of choice without any demonstrable benefit. All they know is IT is a sinkhole of money and an endless source of headaches. It was soooo much easier for them when it was just COBOL. They don't don't have to worry about getting accountants who know the company's choice of competing standards, they can assume everyone has some basic proficiency in the de facto national language. It just pisses them off that the techies can't even agree on what language to use.

3) Corollary of the above. 99% of the people involved are completely incapable of having the discussion.  Once "dynamic languages" leaves your mouth, you've made yourself irrelevant. You're just an annoyance. Sure, "productivity" is on their slide show about the "Blueprint" but they're not capable of drilling that down to details. Their main concern is coming up with frameworks to limit choices, not expand them. (Don't try to teach a pig to sing, it wastes your time and annoys the pig.)

4) If the Accentures and Siebels of the world aren't putting millions into marketing it, it ain't going to happen.

Jim S.
Wednesday, August 20, 2003

Jim, out of curiosity, are those solely the opinions of telco execs, or do you share them?

Philo

Philo
Wednesday, August 20, 2003

Jim, that was pretty interesting, and definitely puts things in perspective.  Unfortunately where I work, we are "technologically-bound" (as in CPU-bound) which leads to some interesting programming practices as well.  : )

Andy
Thursday, August 21, 2003

I still prefer when the compiler reveals my mistakes,
and not the debugger. Whenever possible.
That's why I like strong types.

Writing (correct) test suites also takes a some amount
of time. Writing even more tests to detect errors
within variable types - that a strongly typed language
wouldn't even compile -  doesn't sound like being more
productive.

Christian
Thursday, August 21, 2003

So the compiler checks the original typing of passed parameters from a non-local call?

A compiler is not a tool to validate code its a tool to convert it from one form into another. 

Simon Lucy
Thursday, August 21, 2003

A compiler validates the syntax of the code before
it translates it.
It must not compile invalid programs.
Strong typing is a syntax element.
(Also for remote calls, depending on the technique
you are using.)

Another example:
Some languages do not require variable declarations at all.
Simple typos in variable names remain undetected
until you test the program and run through the critical
code section.

I prefer the compiler showing me these typos.
Other people might prefer not to declare variables,
because coding goes quicker then, and the test suites
would find the typos.

Christian
Thursday, August 21, 2003

Jim,

Thanks -- you're the only one in the thread so far who's actually addressed the question. :)

I think technologies sometimes enter the Fortune 5000 from below, after having been proved in lots of smaller, medium-sized shops. Thus if medium-sized shops can show some ROI from using dynamic languages, the biggest companies will pay attention.

However, I don't as yet see any groundswell here; Monster lists thousands of Java jobs, but only 100 Python jobs, and virtually no Ruby jobs (most of the ones I found were for a restaurant called "Ruby Tuesday"). So far dynamic languages seem to be mostly a techie phenomenon.

Peter Breton
Thursday, August 21, 2003

Interesting...  the company I work for actually works with Accenture and Siebel.  My little 4 person team regularly embarasses them with our productivity.  :)

Truth be told, though, it has little to do with our language choice, it's more to do with our having more business savvy. 

Anyway, one of the systems I designed (Python-based, first written 5 years ago) was able to be rolled out to dozens of locations while the "official" system was supposed to be Vantive...  then Siebel...  and now the entire group that was responsible for those efforts has been outsourced, and the internal customers have all voted with their feet for the system our little group built.

So, does quality make a difference?  Sure, if it's a quality that the users want or need.

Phillip J. Eby
Thursday, August 21, 2003

Phillip - I'm glad to hear that. I have this innate mistrust that Siebel, et al, are just the big brothers of Access - "You don't need programmers with our suite of tools!" I can't help but feel that there's nothing about "Enterprise Applications" that can't really be handled with good design and some competent coding.

And that's not just because I beat out IBM Consulting and built an Enterprise app by hand, either. ;-)

Philo

Philo
Thursday, August 21, 2003

The thing that drives me up the wall about the Siebels and Vantives and other high-priced data models (because all you *really* get is a data model, some screens/reports, and a VB/Access-style toolkit), is that the consultants who drive the implementations are clueless, fresh-out-of college people following The Methodology.  (See Philip Greenspun's writings about that.)

So, you have these implementations of business critical systems being driven by people who don't understand business in the general sense, let alone the specifics of the actual business being affected.  They usually then meet with a bunch of IT people from inside the company, who are similarly unclueful.  God forbid any actual *users* should participate!

And they propose "best practices" that suspiciously mirror the capabilities (or lack thereof) of their own software, but they can't explain what benefit these so-called best practices are supposed to have, because they don't *know*, because they're not business people.  Bleah!

Oh yeah, and that data model you're getting for all that money?  It's usually not that great of a data model.  It's an incredibly convoluted thing that's evolved from hacking on it to meet the needs of miscellaneous clients.  For the specifics of what one's company needs, it'd often be much better to start from scratch.  (Not that I'm claiming my own data models to be any better.)

The software I mentioned earlier, was originally commissioned to replace another one of these idiot behemoths, Clarify.  Clarify has been defunct for a while now, as has Vantive, which we also replaced some installations of.  I expect Siebel will also vanish from this earth before my system does.  And yet, in the past, some IT staffers fought the continued usage of that system because "it's not supported by a company".  It has taken a few years, but enough key management in the company now realizes that this is an *advantage*, both competitively (time to market and deal flexibility) and in TCO.

Phillip J. Eby
Thursday, August 21, 2003

Don't know anything about Siebel in particular, but I had an interview with them a couple years ago when their stock price was like $120 or something.  The guy had been working there for like 8 months, and was a huge arrogant prick, basically taking the attitude that I should be grateful to even have the interview, and they were the best shit since sliced bread, and they were going to change the world, and have this "elite" workforce of expert coders.

All I can say now is HAHAHAHA.

Andy
Thursday, August 21, 2003

Back on  topic, is there anyone, with signficant experience in both static and dynamic languages, who prefers static? Don't respond if you simply dislike Perl's syntax; I'm wondering about Java vs. Python.

Julian
Friday, August 22, 2003

I was a hardcore C/C++ guy, and I tried Python, and liked it very much.  Not sure I am completely convinced it is good for large stuff.  Something about static types is more self-documenting.  And laying out structs and classes seems to help the design process.  Just randomly adding members to classes like you do in Python is a bit disorienting.

The right tool for the right job.  I don't think it is an either/or question.  If you know one style, learning the other one is only going to help.

Andy
Friday, August 22, 2003

When developing larger applications and frameworks, I use interfaces to spec things out.  One advantage to this in Python, unlike Java et al, is that you can use the adapt function (defined by Python Enhancement Proposal #246) to adapt objects from one interface to another.  Sort of like casting in Java - with the difference that the implementation of an interface is supplied by a different class than the object.

To put it another way, if I create a framework interface IDocInfo that has methods to return documentation metadata about a component, and I need to use a class that doesn't implement this interface, then in Java I'm out of luck unless I can somehow get an adapter object into the method that casts to IDocInfo.

But, in a Python program, I need only *declare* that my adapter class implements the interface on behalf of objects of a given type, and any code that calls 'adapt(foo,IDocInfo)' will get an instance of the adapter (unless foo implements IDocInfo, in which case it'll just get foo).  If no adapter is available, then NotImplementedError will be thrown.

I imagine you could write something to do the same thing in Java, but it probably wouldn't be as easy to use the result.

Anyway...  there are times when using a more "static" structure is useful, and times when more dynamic structure is better.  I'd love to see a future version of Python allow optional type declaration for variables....  but really only to do transparent adapt() calls, so I could declare a variable or parameter to be of type IDocInfo, and have the interpreter call adapt() when I assign to it.  That would be plenty enough static typing for me.

All in all, I'd rather be able to explicitly check/transform type when I need to, than have to jump through a lot of hoops to do anything dynamically.  Python makes simple things simple, and complicated things possible, while (IMO) Java makes *everything* complicated, from your directory layout on forward.

Phillip J. Eby
Friday, August 22, 2003

I don't prefer static but it pays the bills.

As for weak typing, look into the mozilla jars and notice how much is written in javascript. Yes all the 'back end" pieces are in c++/xpcom, but all the interaction/ui code is javascript.

fool for python
Friday, August 22, 2003

And this is worth a read. It's argues the "Tests versus Types" angle:

http://www.osteele.com/archives/2003/08/test_versus_type.html

fool for python
Friday, August 22, 2003

And jython is one the the best productivity enhancers for java:

http://www.joeygibson.com/blog/tech/jython

I'm done.

fool for python
Friday, August 22, 2003

I agree with the tests-instead-of-types argument.

But let me play Devil's Advocate:

The problem I've seen is that many large applications do not have tests, so the TDD argument is invalid in the Real World.

In fact, the Big Ball of Mud point of view is that having types for an enterprise application is good, since the compiler limits the damage that programmers can do. What you are in fact looking for is not the most productive language, but the right size straight-jacket: loose enough to be productive, tight enough to restrain incompetent programmers.

The corollary to "enterprise applications do not use dynamic languages" is what an earlier poster said: programmer productivity is not the key driver.

Peter Breton
Saturday, August 23, 2003

Peter, what you said is quite true -- if your company doesn't depend on technology for competitive advantage.

If your company *does* depend on technology for competitive advantage, and you're competing against mine, then by all means use developer-straitjacket languages, and hire lots and lots of developers.  :)

There is a simple reason why large enterprises prefer to use large numbers of interchangeable, mediocre developers: Management doesn't know how to tell the difference between an excellent developer and a lousy one.  That simple fact is all that stands between mediocrity and excellence in corporate software development.

You may say that management doesn't care, but that's not true.  If they knew they could save even 10% on development costs by hiring fewer, more talented developers -- and they were *confident* that they could tell the difference between talented and untalented -- they would do it in a heartbeat.  It is only the lack of ability to be *certain* about a developer's ability that keeps them from doing this.

Phillip J. Eby
Saturday, August 23, 2003

Miscellaneous comments about the above:

Great (10X) programmers outperform mediocre programmers (almost) whatever the language. And, interestingly, people are most productive in the language they already know (certainly in anything but the long term, and how many corporations take a long term view?)

As pointed out above, it is very difficult to judge great programmers during hiring (and often later) if you use traditional techniques. In fact (according to Lou Adler's "Hire With Your Head", and other publications) most hiring has a 50% success rate, so you may as well flip a coin! And of course you can do lots better if you test what you are hiring for.

In examining the choosing of a programming language in corporations, it is well worth remembering Geoffrey Moore's "Crossing the Chasm". Most managers buy what their peers buy. Only the "visionaries" examine critically the choices available, and there are few visionaries!

In project failures, there is usually "no single point of failure" (read Crichton's "Airframe").

Finally, a technical issue. Let's not forget Dr Alan Kay's reminder "Most of current practice today was invented in the 60s". Today we have such aids as Type Inference, wherebye we no longer manifestly type, yet the compiler checks that our data types implement the appropriate interfaces.

Alex Peake
Saturday, August 23, 2003

Since a majority of software projects are poorly done [reference Standish Chaos reports], and a many or most of them are done in staticly-typed languages, this provides evidence that static typing does little to help provide program correctness.

Some software projects do Test Driven Design / Test Driven Development [see writings of Kent Beck, Robert Martin, Ward Cunningham, Ron Jeffries, and others] and they report that they very low defects compared to their previous projects without TDD, this provides some evidence that writing "checked example code" does a lot to help provide program correctness.

The inventors of TDD used Smalltalk - a strongly-typed, dynamically-type-checked language.

The projects currently using TDD are using various languages, including Java, C++ (a loosely-typed, staticly-type-checked language), Ruby, Smalltalk, etc.

keith ray - homepage.mac.com/keithray/blog/
Saturday, August 23, 2003

In C/C++, this code example shows two type errors NOT detected by the compiler:

    char * x = 0;
    bool y = true;
    func( x, y );  // no warnings, no errors

where func is declared:
void func( char * x, int y );

In test-driven development, I get meaningful output related to what I intend my code to do:

Failure occurred in test_multiple_positions: Expected <true> but was <false>

I quoted the above from this web-site (http://jimweirich.umlcoop.net/articles/tdddemo/index.html) instead of firing up my development environment and making one of my 400+ programmer-tests fail.

I program in C++ by the way, because none of the dynamic languages provide the ability to do "twain plugins" and other system-level code easily on three OS platforms.

keith ray - homepage.mac.com/keithray/blog/
Saturday, August 23, 2003

I 've been working in software product development companies for nearly a decade, however I just started working for a telco.

I've got to agree with the observations of Jim S. about the telcos.  Programmer productivity is the least of their worries. 

Scott Adams the author of "Dilbert" was in fact employed in one of these telcos.  Now I can truly relate to where he's coming from.

Carlos Perez
Saturday, August 23, 2003

Just for the record, most people I've spoken to dislike latent typing (strong, dynamic) due to lack of education rather than due to any problems with the typing itself.

Many programmers confuse weak typing with dynamic typing; this is a large part of the problem.

aaa
Saturday, August 23, 2003

*  Recent Topics

*  Fog Creek Home