Fog Creek Software
Discussion Board




Web Application Development Best Practices

I had a question I've been scouring the Web for, and have been unable to turn up anything.  No handle to grep by.  I was hoping the denizens of the JoS forums might be able to help.

My team is attempting to rehaul our development model to include source control and the use of IDEs (don't even ask what we used to do).  We develop web applications in PHP/MySQL, in a Linux dev environment.

We want to move to a client-server model, where the repos resides on the server, and the working copies reside on our personal workstations (clients).  The tricky part is that PHP and MySQL must be deployed and supported at each workstation, so that each developer's working copy can be ran and tested before being commited back to the repos.

Anyone know about a better methodology for this model?  Is there a specific term for this?  I had no idea about what terms to search the web for.  'Web application' & 'best practices'  and the like did not turn up much.

I would appreciate any pointers

Jeff
Thursday, June 24, 2004

I have always used a single dev server with individual sandboxes.  The dev then commit and update their sandboxes as needed.  This works nicely as long as you can handle the remote access issues.

If you are seeking to have clean installs from scratch, it's possible to do the dev on each workstation, but realize you are going to have to coordinate the all the server settings and software. The best way to address this is to create an installer., something few web shops have the luxury of doing.

Sassy
Thursday, June 24, 2004

Advice: forget client-server and focus on n-tier.  Client-server is a dead end from the standpoint of scalability, fault tolerance and performance.

Anon
Thursday, June 24, 2004

Just to clarify,  I am refering to our development infrastructure, and not to our actual software/applications.

Jeff
Thursday, June 24, 2004

>Just to clarify,  I am refering to our development >infrastructure, and not to our actual software/applications.

Can you expand on this? I am not exactly
sure what it means.

son of parnas
Thursday, June 24, 2004

i think the setup you describe is pretty standard. the biggest problems for me are 1) loading/maintaining test data on the local box and 2) the dev/production servers are Novell and the local box is Windows, thus running quite different versions of PHP.

alternatively, could you log in remotely to the dev server, check out working copies on the dev server, and work on those?

josheli
Thursday, June 24, 2004

Realize that you are going to essentially create a big configuration headache by trying to maintain exact copies of server environments on your workstations.  How do you plan to address:

* multiple versions of PHP, mySQL?
* PHP.ini, my.conf ?
* database schema changes?
* web server configs

Sassy
Thursday, June 24, 2004


Three things:

Have a primary production server.

Have a primary development server.

Let your developers have their own personal sandboxes to do whatever they want.

Before they can deploy to the development, they must successfully test (preferably automated) on their own sandbox and do a diff between their box and development.

This diff shoudl include additional libraries, versions, all code, database schema, etc.

KC
Thursday, June 24, 2004

Here's what we are looking at:

*Identical servers (php config, mysql version, etc.) for dev and prod
*using source control, check out files from dev onto local workstation
*optional sandbox testing on local machine, but "official" dev could be tested on local machine's webserver using the real dev server for the datasource.
*when safe on local machine, check it back in to the dev server
*when safe on the dev server, sync dev files to the prod server

You would have to maintain that the database schemas on dev and prod stay in sync, but that's better than a dev db on every workstation.

We have not gotten this set up yet (we've been in guerilla mode up to this point), so I'm very interested in reading feedback to this approach, in case I'm not seeing some potential pitfalls.

Clay Whipkey
Thursday, June 24, 2004

Set up a continuous integration system as well.  Something along the lines of Cruise Control ( http://cruisecontrol.sourceforge.net/ )  There's even a .NET implementation for you Microsoft fans out there.

With CI, you can make sure your system always builds, and you can lay the smack down on the fool that breaks the build that much faster.

joev
Thursday, June 24, 2004

Anyone have a setup they're particularly proud of? 

Another issue to deal with is having to flipping some switch on the config of a working copy to tell this copy of the application where different paths are (i.e. db path/login,  etc.),  and settiing it to a value based on the working copy location and workstation set up.

Jeff
Thursday, June 24, 2004

On review, that was a terrible run-on sentence, sorry.

Jeff
Thursday, June 24, 2004

We perform development on a web-community type “thing” (technical term) using PHP and Sybase ASE. Obviously not Windows development, so this might not be the best for some particular development style.

*DEV server
All sandbox-style development happens here. This is a mirrored copy of PROD but aged considerably (e.g. every few months we refresh DEV). Everyone has almost full rights to access the database from code/SQL, but generally most people don’t do much that would impact someone else (don’t wipe the users table, for example).

Developers can create any/all views/stored procedures/etc. but cannot destructively alter the DBMS schema without the Senior DBA’s approval (via a quasi-committee). Generally, no one really needs to change the table schema very often, so this is no big deal.

*STAGING/QA server
This is a mirrored copy of PROD which is refreshed every time PROD has a check-in event. This is where code, data, etc. from DEV is promoted to after careful code review via the development team and manager (and a DBA if it involves schema changes/SQL, or a Security Admin if it involves exposing new data to users, etc.). Data can only be added/removed via the UI. Here is where upgrade scripts, regression tests etc. are performed and QA works their magic.

*PROD server
This is obvious. Production, live, The Big One, etc. Changes which didn’t blow up on STAGING get promoted to PROD.

*QA/REPORTING server
This is a nightly mirror of PROD for QA and reporting use. Keeps load off of PROD so customers don’t go nuts. It’s also use for statistical reporting and such.

*Source Control Server
This contains the PROD source code with a branch for STAGING. We’re debating whether or not to create a branch for DEV (even when you are developing new code sometimes it’s nice to be able to rollback some changes you may have tried). The software handles merges from STAGING to PROD if/when necessary. Developers check out source code, edit it, and then push it up to DEV to test. We’re toying with local developer copies of the ASE database but will probably not do it for the size of the DB is non-trivial. We’re also evaluating Replication Server to push changes from PROD into QA/REPORTING so that we can have real-time statistics instead of day-old ones.

*PROS*
Since the different environments are separate an errant developer cannot take down the production machines. The processes are rigorous enough and the environments identical (hardware is all the same) that we’ve never had a faulty PROD update (e.g. something was promoted and it broke).

*CONS*
The processes are rigorous enough that a “quick fix” takes longer than it should. As we know from articles (Joel’s etc.) that a main advantage of a web app is that it can be patched quickly. Obviously if there is a really critical bug that needs fixed yesterday the red tape is cut quicker than the red sea, but usually the cycle is long enough that a bug fixed yesterday won’t show up in PROD until today or tomorrow, depending on the winds, how bogged down QA is, etc.

Captain McFly
Thursday, June 24, 2004

In case you're wondering where I fit in the process -- I'm the CTO so I just worry about hiring people that "get things done"  and playing with things I really shouldn't. :)

Captain McFly
Thursday, June 24, 2004

"Another issue to deal with is having to flipping some switch on the config of a working copy to tell this copy of the application where different paths are"

All my PHP applications use an XML based configuration file.  There is the base configuration and then there is per-hostname configuration which overrides anything in the base. 

When the site is in testing, the hostname is "somesite.test" or "somesite.laptop" -- So it uses the test configuration.  The live site has a hostname of www.somesite.com so it reads the live configuration.

I never have to worry about "flipping a switch" to control the configuration.  It's straight xcopy deployment.

Almost Anonymous
Thursday, June 24, 2004

Jeff,

Is there some reason for the operating system choices that you have described?  Novell would be near the bottom of my list for server choices. Ideally I would choose some variety of free or commercial UNIX for the server, or at least a Windows machine running Apache or IIS.  Unless there's something magical about this Novell server that is required for your Dev work, I'd see about getting a real web server (yes, I'm biased; I've never met a Novell server that I didn't hate).

I would synchronize my workstation OS with my server OS.  This makes life a whole lot easier, allowing you to synchronize PHP and MySQL versions.

As far a keeping up your workstation installations, it's a little tedious, but not overly so. One person will need to be selected to stay on top of the current version and enforcing compliance on the workstations, but it's easy to see who's out of compliance with some simple automated scripts.

Clay Dowling
Thursday, June 24, 2004

"My team is attempting to rehaul our development model to include source control and the use of IDEs (don't even ask what we used to do)."

Don't try to add all the processes and tools at once, your team will drown.  Yeah, start with source control (oh boy).  Subversion of course.

Brad
Thursday, June 24, 2004

Clay,  I think another poster said Novell when describing his own set-up.  Not us.  We use only Linux around here.

Brad,  you bet we're using Subversion. 

Jeff
Friday, June 25, 2004

After some rumination, I think we are going to ditch the environment-deployed-at-every-workstation model. 

Now, all working copies will be in different paths on the same file server.  One PHP/MySql/Smarty install and config on that machine.  NFS mounting will enable clients to access these copies thru their locally running IDEs.  We want to leverage client horsepower for the IDE cuz lotta these PHP IDEs are memory monsters.

Jeff
Friday, June 25, 2004

My bad, Jeff.  I associated the wrong signature with the message.  You're in a much more sensible arrangement.

Clay Dowling
Friday, June 25, 2004

Jeff, I am planning a setup similar to yours for a little development project I am working on that seems to get bigger every day.  What IDE are you planning on using?   

Jon Lindbo
Friday, June 25, 2004

Heh, that's opening up a whole new thread.    =)

We're stiill evaluating the different ones.  We are looking at Eclipse (w/ PHPEclipse plug-in),  Zend Studio, Komodo, and NuSphere's PHPEd.  Nothing seems to jump out as the clear winner.    I basically would love an Intellij IDEA for PHP. 

Features I'm looking out for:  a smart global search and replace, code completion, php manual pop-ups,  identation as smart as emac's.    Support for Subversion is not built into any of these (there's a marginal Eclipse plug-in).

What are you looking at?

Jeff
Friday, June 25, 2004

Hey Jeff,

How about using one developmet server and give each developer shell access to it (with a possible Samba share, for those Windows users). In your Apache configuration define a virtual host for each developer (or each application per developer) (with a little bit of scripting this is pretty efficient). To top things off run your own DNS server (named) so that your X app for developer Y runs at the following url: http://Y.X.dev.

The advantage? One Apache, PHP, Mysql, etc. etc. So no differences between developers there, and you only have to upgrade one package when you're going to upgrade.

Jilles Oldenbeuving
Friday, June 25, 2004

The only PHP IDE I have really worked with has been Zend Studio.  I think I am going to spend some time with Eclipse and see if it will fit with my small group. 

Jon Lindbo
Friday, June 25, 2004

Sounds like the Java scenario we are moving to which imho sounds like it sucks big time.

And you will use an IDE for PHP. You gotta be kidding.

Anyhow, what we have now for PHP works GREAT imho.

We have 3 boxes:

1) dev
2) stage
3) prod

and a source code repository that developers use. We have a CMS system running on dev for content and html people.  Every change by everyone is pushed through the CMS. How they call CMS to make the push differs. When something is done you push all your files from dev to stage. Testors then verify what you've done and if it's good to go they push. Everything recorded, everyone notified.

So far I cannot imagine a more simpler system that has the controls that this has.

We are now going to an environement where the SIMPLEST change requires a SITE PUSH. Where each developer has a copy of the SITE on his development box and he must deploy to the X box (not 1,2,3 anymore) ... I really don't see how it is going to work. Seems really ugly and slow and a pain with no gain.

me
Monday, June 28, 2004

*  Recent Topics

*  Fog Creek Home