The Old Joel on Software Forum: Part 3 (of 5)

Writing a software schedule

I've been asked to draft a timeline for a new proposal we're making to a prospective client against a requirement of his. The time I have is today. By the evening, I must have given them an estimate in man hours/man days, working days, calender days and stuff.

I'd like to granularize the function points involved in the project to whatever level is possible at this stage and then draft an esimated schedule. I've seen it, Steve McConnel talks about it in After the Gold Rush, Joel talks about it all the time, Steve McGuire talks about it in Debugging the Development Process, and every other guru preaches that a realistic schedule is only arrived later at an evolutionary stage when you're somewhere in the middle. However, business is business and we *have* to have a starting point to make business decisions and hence I will have to deliver a ballpark estimate of what its going to take in terms of time.

The content of the writing would be based on analysis of the facts I have with me, but I need some "form" I can lean on to. I'd go with Joel's idea of unconventional specs and stuff, but this is the first time the management has entrusted upon me en enriching responsiblity and I want to stay a bit formal since the document is going to be delivered to the client. Is there some place I can get started on the writing so it looks a bit stiff, at the same time it mustn't look like a CMM kinda scary stuff. I hate what I am doing but this is the first impression and I don't want to take the risk of coming out like a flippant kid guy.

Sathyaish Chakravarthy
Monday, November 10, 2003

Joel's *method* is unconventional, but the format you can deliver with a painless software schedule can be very formal.

You break down a spreadsheet into functional areas, then design units, and down until you feel comfortable assigning an hour amount to the task. Make sure if you're assigning a task to multiple people you can justify how the task will be divided between them.

Make sure if you aren't sure how something will be done, you provide estimates for research where necessary. Don't be optimistic.

When you've broken everything down, create subtotal and total columns (WARNING: be absolutely POSITIVE you don't add the subtotals and the line items, effectively doubling the estimate).

When you've finished, create a cover sheet with an executive summary and the unit-level subtotals leading down to a grand total. Attach the spreadsheet.

Hope this helps,
Philo

Philo
Monday, November 10, 2003

Thanks for the lead, Philo. Joel's method is the method to go with; that's the only sensible way I could arrive at some practical figures.

However, I have one more bolder on my head. Like all developers, I am poised and want to collect all the possible information available, apply it as an input to my thought process and take some time to quote a figurative timeline. But management guys are always in a hurry. A few of the guys deliniated responsibility and passed the word down the line, "Go! Get the schedule so we can quote." And they say, "You're giving it today, aren't you?" And now all of them are waiting while doing *other* management things while one guy's thinking his balls off trying not to make mistakes. I know what a wrong judgement can cost later.

In my heart, I don't want to be the cause of wreck at a later stage. I want "time" and "information" to give realistic figures especially since I bore the left-overs of an ailing project sometime back and resurrected it in 4 months time. I want to be looked upon as a sensible guy making sensible estimates, but for that I have to tell the management that it is not enough to commit figures at this stage. I think to this, they will not listen. Their viewpoint might be, "Business must go on, Sathyaish! How do we quote then?"

What is your experience? I am confused. They are right. I am also right. How does business work?

Sathyaish Chakravarthy
Monday, November 10, 2003

Ahhh... this was the second reason I recommended "Pentagon Wars" for software developers.

The subject of interest was the effect of vapors on the troops in an armored vehicle when it was hit by enemy fire. The independent officer assigned to review the vehicle wanted to put live animals (specifically sheep) in the vehicle during a live fire test to see the effect. So the people in charge of the testing set up a "Ruminant Procurement Office" to study what kind of sheep should be used - what sheep most resemble the cardiovascular system of the average Army soldier? Should there be a cost/benefit analysis? Would the sheeps' wool affect the test? And so on and so on. They estimated 6-9 months before they could recommend what kind of sheep to use.

Our hero walked out, went to a farm and bought a flock of sheep, then ran the test.

My point is - you will never know all the variables necessary to conduct a completely accurate schedule. Accept this, and recognize that your best guess (given the thought required for a painless estimate) will be as accurate as anything short of building the thing and adding up the time cards.

Management wants to know how to bid it, so put together a guesstimate that you would be comfortable doing the job for if your paycheck depended on it.

I hope this helps,
Philo

Philo
Monday, November 10, 2003

I get the idea. Thanks!

Sathyaish Chakravarthy
Monday, November 10, 2003

what philo said.

but with one addition -- document your assumptions so they're clearly identifiable.

In our contracts, we have a separate section for assumptions and list them as numbered items, with explanatory text following each as needed. Even if we already mentioned an assumption earlier in our proposal, we'll include them again here just so they're visible.

Since you're having to forge ahead with incomplete info, you're going to have to make assumptions. Don't be surprised, assuming the client even bothers to read the document, if they then come back with some clarifications and maybe even scope changes, because seeing your assumptions will have prompted them to think further about the project than they probably did to begin with.

Good luck.

anonQAguy
Monday, November 10, 2003

When asked to make an estimate early in development, I would argue that it's better to give a guarenteed finish date that takes a long time than a "best hope" date.

http://www.csis.gvsu.edu/~heusserm/articles/BetterNever.htm

But that's just me ...

Matt H.
Monday, November 10, 2003

I'd deliver the estimate as a range estimate instead of a point estimate. Collect both an aggressive (50% likely) and a comfortable (90% likely) estimate for each unit of work. Computing a total with information like this is tricky, if you want to do the math correctly (you have to use statistical methods to get it right), but you can add 'em up as a first order approximation. Show the range to the boss and ask him where he wants to draw the risk line if he really wants a point estimate.

bury 'em with statistics
Monday, November 10, 2003

Whoa! This is what I gave them at the end of 8 pages.

Best case = 10 man months less 20% negative slippage = 8 man months

Most Likely = 10 man months

Worst Outcome = 10 man months plus 20% slippage = 12 man months

They were like I'd given them a harder nut than they could bite. Its a *huge* *huge* project I know and they are still not listening to sounds that don't rhyme with 4 months, which according to me is ridiculous. I padded all the buffers Joel mentions - vaccations/holidays, contingencies, integration time etc. but at very modest levels. Also put in 40% of the planning & development time for debugging.

Sathyaish Chakravarthy
Monday, November 10, 2003

Good luck, bro. Sounds like you're gonna need it.

bury 'em with statistics
Monday, November 10, 2003

Stick by your guns. I am constantly battling the forces of evil...er managment. I recently gave them a timeline for migrating all of our code from classic ASP/VB to .NET. They were none to happy. They have two options they can either give me the time needed or spend money on more resources (programmers). Now the ball's in their court.

Good luck.

shiggins
Monday, November 10, 2003

Jesus! I realized what a blunder I made in writing the estimate. Do you catch it? I wrote:

*man months*

where it must've been months for a team of 4 developers. Or I should've written *40 man months* where I wrote 10 man months. But no harm done yet, they're reading it like I want them to - 4 (they miss the man or woman here) months.

Sathyaish Chakravarthy
Monday, November 10, 2003

Sathyaish - There is some great advice on this board that I have used all the time when giving estimates. In my team, the managers like to give me the job of delivering estimates because I understand what their needs are (I do IT work if this makes a difference). The key is to find out "what problem are we trying to solve" in delivering these estimates. Here are the two problems I have come across:

The Manager Estimate

This is when there is a lot of momentum to get a project done and more often than not somebody has already figured out when the project has to be done by. What you end up getting is a manager stopping you in the hall, telling you they need an estimate ASAP. The reason they are doing this is because they know at some point in software development projects, they need to ask the developers for estimates. I don’t think they do this diabolically, but asking for the estimate early on with little detail of the problem gets that task checked off, then we can all move on and get our due date delivered to us.

When I build one of these estimates, I don’t put a lot of thought into the granularity details. I can whip one of these out in a day or two. It basically involves a high level comparison of this project to similar projects you have done in the past. They key to keeping your credibility is to list all assumptions you are making (as has been mentioned), give it an estimate range (we are allowed 100%, 50% and 20% variance) and most importantly only give durations even when they asked for delivery dates. I have found with just over half of these estimates that by the time I have completed the estimate; a due date has already been published. An exercise in futility. If management is serious about wanting to give adequate time on this project, they will then take your recommendation to let the scope and requirements be defined, then give a revised estimate. This has only happened to me once.

The Project Manager Estimate

This is what I consider a real estimate. The PM is usually trying to make an attempt at putting together a schedule. These are not fun to do because they take a lot of time. Often the PM can be persuaded to give you some extra time to really figure out the duration of these tasks and break down components into assignable tasks. You are, after all, helping the PM tremendously because this is their bread and butter and nobody but you can come up with this information which is so vital to figuring out a completion date. This is where you need to stick to your guns, loose the optimism and find out from the PM what they will use as a fudge factor (“I always multiply my developer estimates by two”).

This estimate should ideally be delivered after the group of developers have reviewed the requirements and have had time to think about them. Meet privately with the developers and talk about who wants to do what. Or, if you are not so democratic, start the work of assigning. Make it clear where the division of labor will occur. Notify them that this is their work and you want an estimate back from them. You too (if you are leading the development) should come up with an estimate for their work. Compare notes and talk it over. Deliver the compiled results to the PM with a task breakdown and include dependencies and all the goodies PM’s dream about at night. You will be their hero and next time they ask for an estimate, they will understand why it took you so long. As stated before, assumptions should be documented, but there should be fewer of them and any assumptions related to the requirements should be handed to an Analyst to amend to the requirements. This type of estimate should usually have a 20% variance, which when combined with the PM’s fudge factor should give decent wiggle room (of course developers can always take the total time allotted for any given task, but that is another story).

Some anecdotes:
- I have had managers get estimates (secretly) from each developer who will be assigned the project, and then they go with the best one.
- I have had managers ask for an estimate, then cut it in half and publish the date. Say I told them it would take six months, they would think I am crazy, make it three months and set it to Feb 1 even though the project won’t start for another month.
- I would say about 85% of the time a due date is delivered in which I have no influence.

Good luck and don't fret too much. Make sure to put in big bold letters at the beginning of your estimate that not enough information about the problem was defined to create an accurate estimate.

m
Monday, November 10, 2003

I will attest to not padding individual estimates and then allowing time at the end for tweaks/bugfixing/etc. without any assigned task, just to catch all of the stuff at the end as accumulated paddding works out well.

Flamebait Sr.
Monday, November 10, 2003

>This is when there is a lot of momentum to get a project done and more often than not somebody has already figured out when the project has to be done by.

Yeah, I've seen that happen to me once before. Just once more I was asked to quote for a small project to read/write audio/video output from a QuickCam camcorder and I quoted 160 hours (it included native compression and I wasn't familiary with Microsoft API for compression). Some US localite quoted 40 hours and bagged the project. I got scolded. The guy who got the project never completed it in 40 hours. He took more than 160 hours.

>What you end up getting is a manager stopping you in the hall, telling you they need an estimate ASAP.

Yuck! How did ya know that? In the hall, yes.

>The reason they are doing this is because they know at some point in software development projects, they need to ask the developers for estimates. I don’t think they do this diabolically, but asking for the estimate early on with little detail of the problem gets that task checked off, then we can all move on and get our due date delivered to us.

You're saying they're doing a ritual? You bet they are. Mostly. Not with this project but the last project I did, the client has since then told my boss's boss, "I'll need Sathyaish's signatures on the proposal and his involvement in the scheduling and...(a lot many things I don't dare mention here)". So for future releases for *that* particular client, he'd better be serious. He'd have no choice.

>When I build one of these estimates, I don’t put a lot of thought into the granularity details.

I ended up doing that when I finally reached page 8.

>I have found with just over half of these estimates that by the time I have completed the estimate; a due date has already been published. An exercise in futility.

Nostalgia! Nostalgia! You're giving me nostalgia.

>If management is serious about wanting to give adequate time on this project, they will then take your recommendation to let the scope and requirements be defined, then give a revised estimate. This has only happened to me once.

And is going to happen to me for the future releases of my last project (not the one I am giving the estimates of currently).

>This is what I consider a real estimate. The PM is usually trying to make an attempt at putting together a schedule. These are not fun to do because they take a lot of time. Often the PM can be persuaded to give you some extra time to really figure out the duration of these tasks and break down components into assignable tasks. You are, after all, helping the PM tremendously because this is their bread and butter and nobody but you can come up with this information which is so vital to figuring out a completion date. This is where you need to stick to your guns, loose the optimism and find out from the PM what they will use as a fudge factor (“I always multiply my developer estimates by two”).

My PM was a soul from the Middle Earth, until before he disappeared from the project. He wasn't the one who'd double my estimates. On the contrary, he'd come to me and reveal feature by feature in a brain fart, "Umm! Cra....ckkk...crac...kkk, my fingures crack! Data Syncrhonization is left. How many days will that take?"
I say, "Days? No no no! You don't understand. How can I tell you an answer to something that I'd need a few days to plan for? Only to tell you how many days, I need about (say) two days."
He'd ejaculate, "That's too much. Two days is too much. I believe the whole thing with the data synchronization will take 3 days."

On 15th July, I remember him relating a psuedo-specification he thought he was writing about a project he (and the big Bs were) was confident they'd finish by the 15th of August. They even made a commitment to a client. I chuckled inside me. Also thought aloud in front of him so he could evasdrop my pitiful grin. Cut...I went back to him on the 18th of August and he smiled, "We've not yet completed the specification fully. There are some changes. The Lotus Notes team (of doughnuts IMHO) is busy with that other thing, they haven't had the time to...". And I'd stand upright in the front of him to have him acknowledge his habitual miscalculation in a somewhat unintelligable grimace. They hadn't even started the project.

>This estimate should ideally be delivered after the group of developers have reviewed the requirements and have had time to think about them.

E*x*actly!

>Meet privately with the developers and talk about who wants to do what.

Unfortunately, for practical purposes, dear friend, I am the only resource I have here.

>Or, if you are not so democratic, start the work of assigning.

I wish I could. I did that when I was new in here, not knowing how much talent bred in the nincompoop camp. Then when it boomeranged, and I learnt about the extra-ordinary feats of the amazing technical accumen these wonder-workers possessed :), I was overwhelmed. I just keep to myself now.

>Make it clear where the division of labor will occur.

I am feeling like making a pun on that now. What do I divide - the labour or the labour?

>Deliver the compiled results to the PM with a task breakdown and include dependencies and all the goodies PM’s dream about at night. You will be their hero and next time they ask for an estimate, they will understand why it took you so long.

http://discuss.fogcreek.com/joelonsoftware/default.asp?cmd=show&ixPost=67900

>- I have had managers ask for an estimate, then cut it in half and publish the date. Say I told them it would take six months, they would think I am crazy, make it three months and set it to Feb 1 even though the project won’t start for another month.

That's so much like my people.

>Good luck and don't fret too much.

Sure. Thanks so much. I enjoyed reading your prolix and it was much wisdom in your words.

>Make sure to put in big bold letters at the beginning of your estimate that not enough information about the problem was defined to create an accurate estimate.

I did, all over the place.

Thanks a million, m!

Sathyaish Chakravarthy
Monday, November 10, 2003

>I will attest to not padding individual estimates and then allowing time at the end for tweaks/bugfixing/etc.

Yup! I did just that. Padded the trail, not the filling.

Sathyaish Chakravarthy
Monday, November 10, 2003

As you get down to doing the "real" plan, which will include the specific sequence of things, keep a couple of things in mind, you probably already know to do these things, but in the heat of things, especially the first time out, it's easy to forget:

1) prioritize, based on the best info you can scratch together, the features so the ones the client most wants/needs will be ready as early as possible. Especially important when there's irrational pressure to shorten your estimates, or even editing of what your estimates were. When reality ends up hitting them in the face and something's not done, you want to try to make sure that whatever didn't get done doesn't include the 'sine quo non' features of the system. I've seen it happen that clients will slip a deadline, or agree to a phased rollout once they see that "yes, it really was a hell of a lot bigger than we thought it would be". They're more likely to react this way, if at all, so long as you can have their important features working -- makes the rest of it look like 'clean up' stuff to them. You can't count on this, depends upon the client and the relationship you develop with them, but something to keep in mind.

2) when you're organizing the task list of the project plan (whether you use ms project, some other PM tool, or just a spreadsheet), try to keep in mind the idea of a requirements matrix. If the task list (and therefore the project plan) is organized so that tasks are defined and grouped into feature sets, then it makes it a lot easier for somebody to take the task list and blow it out into a more detailed list of requirements that will prove to actually be useful for tracking where the hell the project is toward completion (it'll be your QA manager, probably, if he's any good, unless you do it or you have a very sharp business analyst who's able to do it). Personally, I haven't gotten my current shop to this point yet with the task lists/ project plans -- our plans are still pretty useless for actually mapping to specific testable requirements, so we still track progress by getting 'gut feel' percentage estimates of how far along people "feel" they are. This is bullshit, but it's also common. The task list should be constructed so it's directly "mappable" to more detailed requirements that testing folks can say "yep, this battery of tests we just ran were all to test major requirement xyz, which was the last open major requirement under this task, so now this task is complete." It actually takes quite a bit of doing to get a team to this point, or even to get people to understand the utility of it, so you might not get your people to this point for a while, but it should be what you shoot for.

3) don't forget to account for iterations of tasks. ms project and other tools don't handle cyclical things very well, so you usually have to hand-jam them in as you go, but you should allow for, say three (or whatever number seems to be right for your team) build cycles to get a feature set through testing. Blowing this out explicitly in the detailed plan and identifying tasks at this level makes the actual effort more visible than just a nebulous block of time during which code cycles between your test team and the devlopers. It helps you see what's actually happening (and later what actually happened).

anonQAguy
Monday, November 10, 2003

Here's another option: Try not to do plan-driven development, and instead do extremely short iterations (two weeks). The content of every iteration is determined at its start by the customer -- they get to pick from a buffet of small features, all of whose effort has been estimated by the development team and none of which will take longer than the iteration to fully implement. The result of every iteration is software ready to shift with only those features.

Read "Extreme Programming Explained" and some of the other Extreme Programming literature for details. Also see http://c2.com/cgi/wiki?ExtremeProgramming since that's where many XP practitioners hang out.

Chris Hanson
Monday, November 10, 2003

>Read "Extreme Programming Explained" and some of the other Extreme Programming literature for details.

I did.

http://www.custom-code-factory.com/forum/topic.asp?TOPIC_ID=157&SearchTerms=Extreme,Programming

I do not blame this attitude of churning out bold and brilliant ideas as advise but what you're forgetting is, the client is "not yet" a client. It is a stage where we're submitting a proposal. Secondly, in the academic world, when we read of extreme programming or any other methodology for that matter, we assume the application of the tenets of the methodology in a scenario where it is taken for granted that we have a control over our environment. This is not always true.

Sathyaish Chakravarthy
Monday, November 10, 2003

Sounds like you need an Extreme Client! :)

m
Tuesday, November 11, 2003

I think Extreme Programming has a lot of good ideas, but I don't think management in most places is ready for bi-weekly reviews. They want to "set it and forget it." For all their "pro-active" talk, they're pretty passive and really want you to take care of everything for them.

www.MarkTAW.com
Tuesday, November 11, 2003