Fog Creek Software
Discussion Board




Drowning in "Process"

It seems the most pervasive software engineering debate today can be basically boiled down to "order" vs. "chaos".

On the side of "order", you have UML, thick spec documents, language coding standards, inter-company communication standards, usability standards, etc.  These are all "good things".

In the "chaos" corner, there's agile process, iterative design, informal discussions and customer focus groups.  A lot of "seat of the pants" coding.  Trust and empowerment are givens at lower rungs of the hierarchy.

My question:  How do companies that are in the "order" camp get anything done?  With 6 productive hours a day, a developer is expected to be proficient in multiple languages and platforms, communicate clearly with other developers, have reasonable domain knowledge, and get something done.  This is difficult, but manageable.

Throw in UML, source control, rigid documentation standards, remote employees, progress reports, gantt charts, DB modelling software, code reviews, change request approval chains, function point analysis, CMM and/or ISO compliance, detachment from the customer, etc.  How do people do it?

My intention is not to mock "process", but rather to fulfill a morbid curiosity about whether companies actually find heavy process useful and economical or whether it's the domain of $100M failed government contracted projects.

Obviously, much of this revolves around the size of the project and the tenure and quality of the developers.

If "process" is thought of as a generic noun, do people on this list feel they have

a.  Too little "process"
b.  The right amount of "process"
c.  Too much "process"

Why?

Bill Carlson
Wednesday, April 23, 2003

I have experienced it along the entire spectrum and here is what it boils down to: You require as much process as can be put in place to offset the dollars spent doing it.  The age old cost/benefit analysis. 

Simple and extreme examples:
NASA every line of shuttle launch code is tested, EVERY LINE.  Every line is documented.  Before any line is changed it is run through process engineering and reviewed by the entire team.  It is them full spec'd out and documented.  Then it is written down.

Why?  Because dropping a shuttle on Miami is a very bad thing. Aside from what happens to Miami, NASA will spend millions to recover from the crash.

Opposite example:  I was called in to maintain a system that had been built three years prior.  None of the original developers were with the customer any more and the last guy left the week I arrived.  Why?  300,000 lines of code and not a single line of documentation in any form.  ZERO.  People think I am kidding when I say this.  I have no idea how anyone could build such a system without a single scrap of documentation in the programs. Further, all the specifications where done via email.  So everything was either a "new requirement" or a "bug fix" the difference being how long they could afford to wait.  With zero process management in place, nothing ensured even the most basic programmer competence. 

I stayed with the system until it shut down.  It paid very well because they feared if I left they would not find another person familiar with the language to take it over.

Mike Gamerland
Wednesday, April 23, 2003

Bill, did you get the memo about the TPS reports?

Mrs. Robinson
Wednesday, April 23, 2003

Consider the exact opposite of heavy process: absolutely no process and a pathological lack of any process except bullsh*t rituals that some undereducated local "god"/despot considers essential.

I posted today on this syndrome. Right now I am working for a place in which:


- The owner conducts all programming and only delegates rote work to his employees. Employees are "code techs".
- The employees have no clue about configuration management or source code control. It would "interfere with their work".
- The owner isn't even educated in a technical area yet considers himself a demi god of SW design.
- It's a routine application software product (basically data entry forms, a user interface) that in terms of internal coding practices resembles embedded, hand crafted code with a million little funky twists. The owner and his coders "have" to be big studs and maintain a fragile, time intensive set of code written at a very low bare metal level.
-  The owner routinely keeps the only copy of source code on his laptop and takes it home whilst the rest of the company is knee deep in work in that exact area.
- Email is answered in a "tag team" fashion on a public computer in a common area because the owners believe that viri will leap out from the computer onto the corporate network if uncontrolled email use is allowed. So you can never be certain if someone in particular sees an incoming email.
- Only the owner can write anything considered a 'spec' and have it read and considered. Any specs, even highly informal and essential writeups, essentially go unread and amount to being tossed down a "1984" style "memory hole". It's almost subversive for anyone except the owner to be heard.- Code changes rely upon cooperative editing of a source file that everyone shares. Does it get messed up? Well, what are you doing this weekend and oh it REALLY isn't as important as repairing damage done because everyone is sitting in the middle of a clusterfuck that the owner mandates as a "character building" exercize. And you should have worked HARDER and just been MUCH more careful because we **have** to share this source code, we just have to.
- The owner/architect promulgates his own whacky local bullshit software jargon that owes absolutely nothing to any widely accepted CS practice.

Now, would you rather have this? (yes, the company makes money despite itself.)


What I'm saying is that process is great. And empowering employees is also great. But EVERYTHING IN MODERATION. 

The "Joel Test" is a great middle ground.

Nasty Curmudgeon
Wednesday, April 23, 2003

"- The owner isn't even educated in a technical area yet considers himself a demi god of SW design."

Hey Curmudgeon, do me a favor - define "educated"

Philo

Philo
Wednesday, April 23, 2003

Bill -
http://www.computer.org/software/so2000/pdf/s2011.pdf

You do what you feel you need to do to get the job done. You don't write a requirements document to have a requirements document; you write a requirements document to get your brain around the problem at hand. If you're on contract you also write it to protect your backside against scope creep.
You do use cases so that you and the users are on the same page about how they do their jobs.
You create state and process diagrams so you understand how the "flow" is going to work, and what states need to be provided for.
You create class models and ER diagrams to translate all the preceding into something you can work with when you sit down to code. If you're on a team they also serve as talking points, and to ensure everyone is (once again) on the same page.

You do not do these things because they need to be done. You do these things because you need the process and product to organize your thoughts and create a development strategy.

They are not goals, they are tools.

Philo

Philo
Wednesday, April 23, 2003

To say Agile Methods totally removes Process is wrong,  according to people like Martin Flower http://www.martinfowler.com/  . Also this document from Mr. Fowler goes on to explan the use of documentation and process in an Agile Methodology: http://www.martinfowler.com/articles/designDead.html

I tend to agree with the idea of 'right sizing' process.  There are few projects where full, heavy wieght processes are needed but they do exist.  But I have not seen a project that can not benifit from some source control and defect/new feature request process.  A consistant documentation process is also nice (even if it is to javadoc major interfaces).  The more people a project gets the more process and communication is required for it even to hope to function at the end.   

A Software Build Guy
Wednesday, April 23, 2003

I'm currently working on a small simple list management application, I've written the concept document, the physical design doco, the technical design doco, distributed all three documents 7 different people for review, all in all it's taken 2 weeks to get overall agreement. Today I got approval to start the build, but guess what? I've already finished and I just havent told anybody, so looks like I'll be getting paid to read JOS and basically goof off for the next 2-3 weeks.

God bless em, thankfully it's not my money (it will be soon though) :-)

Realist
Thursday, April 24, 2003

Oh, and of course the business requirements documentation

Realist
Thursday, April 24, 2003

I think the best writing on the subject is Steve McConnell's 'Cargo Cult Engineering'

http://www.computer.org/software/so2000/pdf/s2011.pdf

He distinguishes between Process-orientated and Commitment-orientated development, which roughly relate to what you describing.  To quote:

    When used knowledgeably, either development
    style can produce high-quality software
    economically and quickly. However,
    both development styles have pathological
    look-alikes that don’t work nearly as well
    and that can be difficult to distinguish from
    the genuine articles.

Ged Byrne
Thursday, April 24, 2003

You need some kind of balance. For developing new products, a more agile approach is needed because you have to experiment.

The Real PC
Thursday, April 24, 2003

I wouldn't say that the agile methods are in the Chaos camp. From what I have seen of agile methods, they are internally very disciplined and ordered, XP in particular. XP is a process designed to get maximum bang out of process buck. Dispensing with expensive processes like documentation in favour of continuous refactoring and tight communication (regular short meetings). XP is a very specialised process tool that is not widely applicable. It requires high levels of developer competence and a lot of buy-in from individuals that traditionally procrastinate a lot (higher management, external departments, customers).

Process is risk management. You do analysis to identify the workload and difficulty of a project, and therefore make its schedule more accurate and predictable. You spec out projects to get your requirements. You perform defect tracking so that you know the exact state and quality of the software you release and are in a better position to protect yourself from costly in-the-field bug fixes. You do configuration management because software IP is really expensive and it would be a shame to lose all of your assets or their history.

I have generally found that the companies who do not practice even basic amounts of process are pretty much stocked with B & C list talent, and inexperienced graduates who had the potential to be A list material but were never mentored right. Very sad really

Richard
Thursday, April 24, 2003

What is the biggest cost for a project?  Maintenance.

Big Up Front Design may _seem_ slower at the outset, because you're not writing code right away.

It isn't any slower in reality though, you reap the benefits throughout the life of the project.

That said, I don't believe every project _needs_ Big Up Front Design.

amp
Thursday, April 24, 2003

I've worked at startups with varying degrees of process. In my experience, the projects that made good use of an organized process (design docs with some UML, code reviews, source control, project plans, brief team meetings each morning, automated unit tests, a bug database, etc.) were more successful and better able to handle the arrival and departure of developers than the ones that didn't.

In particular, I've found that process can make it easier for me to understand my co-workers' code because by the time I get asked to fix a bug in it I've seen the design and reviewed some parts of the code. Because we've agreed up front on the overall design, most of our code looks pretty similar regardless of who wrote it. I can use unit tests to verify that nothing broke and see explanations of previous changes in the bug database and/or source control system. That's much less frustrating than getting an e-mail saying "hey, this doesn't work and Jim's on vacation, can you fix it?" and having to pray that Jim put some comments in that part of the code.

Beth Linker
Thursday, April 24, 2003

Ha ha ha Realist, that's the spirit!

Apropos process and "software demi-god bosses" I once was in a simillar situation: I was doing the programming and my boss was the "architect". This meant that I was free to structure my code at will, but he had strong opinions about the user interface and general behaviour of the program. The problem was that from time to time, he got ideas. He used to come and gleefully announce, "you know, I've been thinking about this a lot, and I think that we really should make the panel a lighter shade of mauve..." Some of these ideas took 3 secs to implement, others took restructuring of the code. Since most of them took 3 secs, he didn't understand how this last idea would take one whole week.

The solution was that I started to implement _everything_ in at least three days. With status reports and all that in the meanwhile. Lighter shade of mauve? Three days. Yes, it was not as simple as we thought. Add "are you shure" dialog? Three days. Etc.

The results where spectacular. He got used to the idea that things take time, and stopped having new ideas every day of the week. So the software finally got finished, somehow.

Dimitri.
Thursday, April 24, 2003

I have two fairly simply goals in software:

1. Ship quality software that meets my customer's needs.
2. Be able to do it again.

Even if a company is able to nail #1, they often get tripped up on #2. Death marches, code hacks and dictatorship type management can often get a product out the door, but you probably can't do it again that way.

Having some degree of process or organization goes a long way towards ensuring that you don't need to resort to death marches to get the product out the door. Not having death marches or <insert bad practice here> goes a long ways towards being able to duplicate your success.

As others have said, having a process isn't the goal itself. Shipping software is. If a particular process doesn't work for your team or your type of product then give it the ol' heave-ho.

I use to be enamored of things like RUP or CMM Level 5 certification. I've seen realized that for most commercial software projects, that level of process just doesn't have a return on investment.

That doesn't mean that process is bad. It isn't. If a process helps you ship quality software faster and keeps your team together, and it can be implemented in a cost effective manner, then it's probably a good idea.

I don't think that a company has to subscribe 100% to any particular process methodology. I like XP, but I don't agree with everything they say. I like Agile, but much of Agile won't work in my environment. I can apply parts of different methodologies in a way that makes sense for my organization and for my team.

Mark Hoffman
Thursday, April 24, 2003

Agile project management != chaos
Lightweight project management != chaos

Our agile process has a very formalized set of steps. It's not anarchy. Anybody who's ever done it would know that.

Brad Wilson (dotnetguy.techieswithcats.com)
Thursday, April 24, 2003

*  Recent Topics

*  Fog Creek Home