Fog Creek Software
Discussion Board




Python for a commercial web application?

I need good email capabilities for a commercial web application (still in planning) which heavily relies on email manipulation. So far it seems as if Python is the best bet. It has better email handling capabilities than PHP and ASP.NET.  Python also seems easy. It's amazing how quickly one can learn and become fairly productive with Python.

Python is cross-platform. It isn't considered as a Linux-oriented or Windows-oriented environment like PHP and ASP.NET. But Python lacks a decent equivalent of ASP.NET or PHP. It seems as if CGI is the only safe option for a commercial web application.

I've found about Spyce, PSP, Zope, etc., but still not sure which one of them is mature and reliable enough to be bundled with a commercial web application. Zope is confusing and badly documented.  Setting up Zope with a web server also seems hard.

Has anyone ever done web development with Python (or any kind of development for that matter)? I'm looking for available options (within Python) which should be practical for a commercial web application. If I am all down to CGI, any suggestions on how to proceed? Build a framework on top of CGI?

I'm looking for feedback.

Rumble Jumble
Thursday, July 15, 2004

By CGI I actually meant to use Python for CGI. Do not take Perl for granted. :)

Rumble Jumble
Thursday, July 15, 2004

I took a look at frameworks for Python last summer, and here are those that seemed worth it as a lighter alternative to Zope:

http://www.cherrypy.org/
http://jonpy.sourceforge.net/
http://www.webwareforpython.org/
http://www.cheetahtemplate.org/
http://www.mems-exchange.org/software/quixote/
http://www.twistedmatrix.com/
http://skunkweb.sourceforge.net/

Fred
Thursday, July 15, 2004

PHP may not have decent email handling capabilities inbuilt but there are several add on modules that do a good job. 

Perhaps you can code the core processing in Python and shell to it from PHP.  That's a fairly common situation.  Or run a Python server (always running) and do IPC to communate with it from a web app.

BTW, PHP is very Windows compatible -- I do all my daily development on Windows XP and the same code is deployed on Linux.

Almost Anonymous
Thursday, July 15, 2004

If you're targeting Windows there's also Python COM. Strange as that sounds, COM with Python and the win32com module is really easy (hey, it's Python :^)

Tom H
Thursday, July 15, 2004

PHP supports COM on Windows as well.

Almost Anonymous
Thursday, July 15, 2004

About four years ago, I started a medium-sized project (~5 KLOC) in Python because I had found the language so efficient when doing some one-off scripts.  Until then (and even now), my primary development language had been Java.  I was impressed by how small "hello world" was and how much less code was required in a typeless (well, at least weakly typed) language.

However, by the time I was done, I wished that I had used Java intead.  Here's why:

1.  Strong typing is your friend.  The project was a macroprocessor for a shorthand XML rendering language for use by non-technical domain experts who authored application metadata.  So, it had to do many of the things required of a good compiler including telling the user what line "caused" an error.  Because of this, I had to drag around token objects that contained source file names and line numbers.  By the end of development, I had wasted hours on bugs resulting from accidently treating strings as tokens.  Debugging required manually inserting Hungarian notation for the purpose of manual type checking.  Java would have avoided all these bugs with little time overhead.

2.  Class member methods were not well enforced in Phython.  I can't remember all the details of this one, but I do remember a class of bugs created by forgetting to include "self" in method calls.

3.  Lots of runtime errors.  Yes, it's hard to write a Java class that will compile.  Hoever, a picky compiler/lanaguage tends to trade compile errors for runtime errors.  On this proiect, I spent a lot of time tracking down runtime type conversion errors and other ways that Java doesn't allow one to shoot himself in the foot.

Recently, I have written a couple of "scripty" programs in Java and wished that I could have used Python (We limit the number of technologies in use on our project to improve the truck factor).  One project did some XML parsing (make a DOM tree and execute a couple of XPath expressions), lots of FTPing (logs on another platform), and rendering of an Excel spreadsheet (Apache has a nice Excel generation API).  For this project, Python would have been more efficient.  It wasn't really a production script, so reliability wasn't as important as developer efficiency.  And, Python has great libraries for XML and network stuff (not sure about MS file format generation) . 

So, in a nutshell, I would use Java for larger, production systems and Python for smaller stuff where time minimization is important.  I'm sure that some .NET language has the same properties as Java, and PERL or whatever is as friendly as Python.  However, I know neither of these; I can only reply to the poster with my experiences.

On the topic of lightweight frameworks, I think that J2EE is probably overkill for most projects.  However, Java has some lightweight frameworks (Avalon and pure Servlets come to mind), and I suspect most popular languages have their share as well.  When I began the project referenced above, I did so because of a cute little Open Source parsing framework.  In retrospect, I would have given the appropriateness of the lanaguge more weight in comparison to the available toolsets.

That's my two cents.

Steve
Thursday, July 15, 2004

asp.net has some great mail handling capabilities:

check out http://www.systemwebmail.com

if your e-Mail is DB intensive, maybe you can let the database handle it (xp_sendmail)

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/tsqlref/ts_xp_aa-sz_6hbg.asp

code monkey
Thursday, July 15, 2004

I've heard that as a disadvantage of Python in large programs, i.e. it is easy to get into situations where you have bugs that you think should be caught by the "compiler".  I do see that as a potential problem -- it almost lets you do anything before even thinking.  Python is not very "strict", it is more like C in that it lets you do whatever you want.

The standard answer is to use stuff like PyChecker or PyLint for more mundane errors, and then unit testing for more complicated errors.  I am writing larger and larger things in Python -- I'll see how that bears out.

One difference between programming Python vs. C or say Java is that I run the program much more often while developing.  In C it is sometimes a bear to compile the program and set everything up, so the tendency is to just write as much code as you can before you run it.  Later you can step through all your code in the debugger.

With Python I have gotten in the opposite habit -- write as LITTLE code as possible before running the program.  That tends to really cut down on silly errors, and also catches bugs as early as possible.

Roose
Friday, July 16, 2004

The python frameworks listed above are good alternatives (TwistedMatrix, though somewhat hard to grok for many people, is especially good).

You could use mod_python if you use Apache, and save the CGI loading overhead if you insist on going CGI.

Ori Berger
Friday, July 16, 2004

OK, so I don't do Python, but I have been writing ruby code for 4 or 5 years . Ruby is another dynamically typed language from roughly the same stable, and which is possibly even more pleasant than Python.

However, I think that teh poblems that the poster mentioned about Python vs Java are because you have to get your head round the fact that dynamically typed and statically typed languages behave differently. Static typing is sometimes your friend, and sometimes your enemy.  I think it is more a question of how your development cycle works - if you are expecting the compiler to catch things, then dynamically typed languages look very sloppy. If you are more into a unit testing type style of development, then the errors fall out in the unit tests.

regards,

treefrog
Friday, July 16, 2004

"truck factor"

Who he?

Rob VH
Friday, July 16, 2004

I might be way off here, but isn't PHP also dymanically typed?

Guyon Morée
Friday, July 16, 2004

PHP is dynamically typed, yes.

Python web frameworks - there are a lot of them about. Zope seems to be the biggest/fanciest/best supported of all of them but I've found it so incredibly over-complicated that I just can't be bothered. I would suggest just using mod_python and its Publisher apache handler which maps URLs onto python modules/functions for you in a secure way. Very simple and straightforward and does all the usual CGI form variable stuff for you, and I think sessions too. A minimum of fuss to get started with. There are various templating systems you can use with this - again I'd probably pick the simplest of them as some of them try to be a bit too much for their own good, especially the zope offerings.

I've not found the dynamic typing to be an issue - although it does make it more important to do unit tests if you're writing complex code. I find the whole advantage and power of Python is the dynamic way it handles objects, classes, functions etc.

Infact I think there might even be some kind of python module/extension that does run-time type checking for you. Even if not you could always just throw in things like

assert( type(foo) is Integer )

where necessary (syntax/keywords probably wrong here but you get the idea), if you want the best of both worlds, dynamic and static typing.

Matt
Friday, July 16, 2004

Truck Factor == # of Employees who must be hit by trucks before a company looses its ability to function.

For some new project, let's presume that an architectural design "optimally" calls for 27 different components/languages/tools/etc.  When I say optimally, I mean that that this set of tools could be employed by developers experienced in their use to produce the system in the most reliable, maintainable, and cost effective manner (among other potential benefits).  The problem is that most developers available would not be experienced in more than a few of these technologies.  So, we must either spend time training existing developers on new tools or hire a whole bunch of new ones.  And, even if we hire a bunch, we are still stuck with areas of there software where only one developer can maintain them skillfully.  If he gets hit by a truck (quits, whatever), we are in trouble.  Also, if a particular phase of development is rather intensive in that skill area, that developer becomes the critical path on the project plan.

For these reasons, we prefer to constrain our toolset to a quantity that results in a more global optimum--Even though we might waste time writing scripts in Java that "ought" to be written in Python/PERL/whatever, we still win because over 75% of our development staff can modify them if necessary.

So, the truck factor really has little to do with trucks....

Steve
Friday, July 16, 2004

"""By the end of development, I had wasted hours on bugs resulting from accidently treating strings as tokens"""

Didn't you define your interfaces beforehand?  Many large-scale Python software frameworks offer interface facilities for designing, declaring, checking, and adapting to interfaces, including PEAK, Twisted, and Zope.  (And PEAK's interface system is separately available as PyProtocols.)

No, it's not compiler type checking, but all the compiler's doing is forcing you to do something that you ought to do anyway: design and document your interface decisions.

If you were using the interface systems of any of the above-mentioned frameworks,  you'd simply do something like:

      def something(self,token):
            token = IToken(token)
            ...

and then if you passed a string by mistake, you'd either get an error, or if you'd defined a reasonable conversion from string to token, you'd get a token.

Python does indeed give you enough rope to shoot yourself in the foot, but it doesn't force you to abandon type discipline where it's useful.  Also, unit tests, e.g. via doctest or PyUnit are especially helpful in building very complex systems.  I recently wrote nearly 3000 lines of Python to create a rather complex expression compiler.  There were about half a dozen interfaces involved, and hundreds of lines of tests of the form:

    self.assertEqual(parse("x>y"), Compare(">",x,y))

During the whole process, I think I went into the debugger maybe only 3 or 4 times, in order to figure out why a test wasn't working the way I thought it should.  Most of the time I just wrote tests, then added functionality to make the tests pass.

Anyway...  it's possible to write very large systems in Python, and I would always reach for Python first, because of that saying: "there are two kinds of large systems: those that grow from smaller ones, and those that don't work".  It's easier to start the growing with Python, and it's very easy to keep the system stable as it grows, if you start it out right.  These days, the only code I write without interfaces and tests is prototype or throwaway code.

Phillip J. Eby
Friday, July 16, 2004

"""Has anyone ever done web development with Python (or any kind of development for that matter)? I'm looking for available options (within Python) which should be practical for a commercial web application. If I am all down to CGI, any suggestions on how to proceed? Build a framework on top of CGI?"""

I've done web development in Python for a bit over 7 years now.  There are plenty of options.  I personally like FastCGI for highly-scalable, actively-developed applications, because it's:

1. Separate from the server process, so restarting an app doesn't require restarting the server.

2. Can be throttled as to the maximum number of requests being simultaneously processed, so as not to blow up the server's memory requirements

3. Application processes can be located on different physical servers than the web server, allowing processor scaling.

4. Allows long-running processes, so you can keep database connections open, caches loaded, etc.

5. A defined standard (see http://fastcgi.com ) with implementations for many languages and web servers

Of course, not all Python frameworks support FastCGI, and some have their own replacements like "pcgi" and "scgi" and the like.  I don't really trust the replacements because they often haven't dealt with support for things like response streaming and "server push", which are occasionally useful to have.  But obviously you can make your own decisions there.

I also have a strong preference for "object publishing" approaches to web app design; this approach is rare in langauges other than Python, but it shows up in several Python frameworks, as it was originally developed for an ancient (circa 1996!) predecessor of Zope and inspired many imitations.

Basically, the idea of object publishing is that URL paths represent traversals of some objects you define, like 'Invoices/33567/view', to view invoice 33567.  You might also have 'Invoices/33567/editForm' to view a form that lets you edit it, which then posts to 'Invoices/33567/change' to actually change it.

In other words, traversal looks up objects or methods, and the last item in the URL is called, passing in the data from the form or query string, if any.  The called object then returns a string, stream, or response object to be sent back to the user.  It's much higher-level than CGI, yet not nearly as complex as Java "MVC" frameworks.

Phillip J. Eby
Friday, July 16, 2004

*  Recent Topics

*  Fog Creek Home