Fog Creek Software
Discussion Board




Software development not NP complete?


I've recently been wrestling with a project I'm finishing up. We expected it to take about two weeks, and it took six - the development path was a mess of dead ends and thrown away work ("can't get there from here" type stuff).

At this end of the process, I'm mortified that I made such a poor estimate; yet walking the path in a forward direction, I can't think of any "bad" decision I made with the facts at hand at the time.

This strikes me as the classic NP-hard problem - when you have the problem and need an answer, there is no way to determine the absolute answer or even predict how long it will take to find it. On the other hand, if you *know* the answer, then it's possible to verify that the answer answers the problem in a defined length of time.

I think this is why estimates are, at best, guesses. Maybe a better approach to the whole thing would be recognizing this?

Philo

Philo
Monday, July 14, 2003


Well... if you default an estimate to mean the same thing as a guess, then you discount the value of the estimate.  Estimates are good to know if you are on or off track during a project... if they are all SWAGs... then you ain't got a clue as to where you are.

I think you have it though when you said "with the facts at hand at the time".  In addition to facts, assumptions are also involved.  As you travel through the project, the learning experience should increase the level of fact and reduce the level of assumption.

Joe AA
Monday, July 14, 2003

You can only really start estimating accurately when you start recording how long it takes you to accomplish a certain type of task. Without a historical record you are just going to be guessing.

When you are doing something new (to you) or you don't know what you will be doing (because the requirements aren't defined) then you don't have any hope of estimating accurately.

Tony E
Monday, July 14, 2003

"We expected it to take about two weeks, and it took six"

Because, you've spent the other four in this forum :)

Evgeny Goldin
Monday, July 14, 2003

Estimation is tough, and often asking the impossible question "How long will it take me to do something I've never done before?"

Joel's "painless software schedule" is a pretty good technique though, simply make an exhaustive list of everything you have to do and add it up. It sounds too simplistic until you try it and realise that like brainstorming you think of all these tasks you overlooked.

Matthew Lock
Monday, July 14, 2003


Well, XP has the concept of "spikes". Spikes are something you do when you have a task you can't estimate with any confidence. The focus of the spike is not to produce code, but to increase the understanding of the tasks by the team to the point where they can estimate with confidence.

The fact that they had to produce a separate concept for this indicates that it's not an uncommon problem.

DingBat
Monday, July 14, 2003

q.v. Turing's 'halting' problem?

constructive comment
Monday, July 14, 2003

I guess you do much the same as I or anyone with experience does, you look at the nature of what's wanted, the current input and output forms or processes think.... hmmmm I've seen this and that kind of thing before, I don't know how we get from P -> Q but it can't be too much...

And then you discover a whole squirly mess in that gap which also affects everything else and you find the basis on which you were working was completely erroneous.

Most often those things are domain specific and only because they've been doing something since Cain thumped Abel so you have to rely upon extracting that domain knowledge they aren't even aware of having.  Sometimes you get it, sometimes you don't.

Simon Lucy
Monday, July 14, 2003

http://www.joelonsoftware.com/articles/fog0000000245.html

   
Monday, July 14, 2003

Regarding painless software schedules - the task was writing a modem pool handler in C#.

Would you have scheduled a line item for "Can't access a modem from terminal services"? (8 hours of research, probably 20% extra testing time)

How about "discover that using terminal services while a modem is active causes a core dump"? (3 days, another 10% on testing)

Then there's "find out the component vendor was lying - their comm component can *not* be used from .Net"? (2 days)

etc, etc, etc

Even Joel pointed it out in one of his articles where he was writing an FTP utility. I'll wager if he'd done a painless software estimate he wouldn't have put a line item in for "discover that MS left a bug in their FTP implementation, research FTP protocols and API's for workaround"

I think a lot of devout estimation/scheduling advocates think of software development as driving from New York to Los Angeles with a map and GPS, when IMHO most of the time it's being in New York and being told "Los Angeles is that way [gesturing westward], how long will it take you to get these twenty elephants there?"

Philo

Philo
Monday, July 14, 2003

If you can't give an accurate estimate, then you are still in the research phase.  Research until you can give a solid estimate, or flat out tell the clients you don't know what you are doing and will have to experiment (worded a little more client-friendly, of course).  It may be that research time takes more time than actual development.  This is the price your clients pay for hiring somebody that had never done this type of work before. 

    
Monday, July 14, 2003

The problem being that "this type of work" can often be incredibly narrow.

"You want a web application built to your specifications against your hardware and your database? Well, unless you can find someone that's already written a web application built to your specifications against your database then I'll be winging it"

"Research may take longer than implementation"
a) Research project, give estimate, bring project in on time and budget. Research budget: $50k. Project budget: $75k

b) Do minimal research, start work, hit roadblocks, regroup, etc. Run 50% over budget. Research budget: $0. Original project budget: $50k. Final project budget: $75k.

I think a lot of problems that people have with projects going over budget is that they feel "stung after the fact." What they don't seem to realize is that getting a valid reproducible estimate may cost more than the job itself. Do people really prefer the $125k cost in (a) to the $75k cost in (b)?

Philo

Philo
Monday, July 14, 2003

wow.  it must make your job easier being able to pull numbers out of the air like that.

the project doesn't have to be the same, only similar. 

Here, let me make up my own numbers (a little more believable):

"I've never worked with that file format before, but I've worked with many others and realize it usually takes 2 weeks to get comfortable with a file format of that complexity, I'll budget 3".

     
Monday, July 14, 2003

I am not sure if it was Ged or Matthew (my apologies if it was someone else), in this forum: Their rule of thumb was come up with an estimate and then multiply that by 4.

Prakash S
Monday, July 14, 2003

So blank, what are you getting at? Are you saying they shoudl have hired you, that you have written a modem pool handler in C# similar to what they were looking for? Or are you saying you know someone who has done this who would have been a better hire than Philo? It seems that unless you know there is a surplus of modem pool handler C# writers available for hire, I  wonder what your point about how they should have hired someone with experience in the area of C# modem pool handlers.

blankety blank
Monday, July 14, 2003

Philo,

"Do people really prefer the $125k cost in (a) to the $75k cost in (b)?"

Yes, basically, I think they do. Most businesses would prefer to know how long something will take and how much it will cost, even if it seems more at first.

The problem with option (b) is that it may be considerably more than $25k extra required and going over-budget (annoying) usually means putting back delivery times (very annoying).

I appreciate that many small companies will try to go for a cheaper option, especially if someone offers it to them, but most bigger companies, IMHO, will prefer to spend more and know what they're getting.

No guarantees, obviously. Even with the $50k research, you might come across unforseen problems, but at least you can go to the board and show 'due diligence', etc.

Steve Jones (UK)
Monday, July 14, 2003

[qvote]
Here, let me make up my own numbers (a little more believable):

"I've never worked with that file format before, but I've worked with many others and realize it usually takes 2 weeks to get comfortable with a file format of that complexity, I'll budget 3".
[/qvote]

Good example. Now let's say the file format you've worked with is X12 850, and the one you need to develop for is X12 810. No big deal, right?

Oh, look - X12 810 allows for Allowance codes, PO Numbers, and Packing Slip numbers at the item level. In 850's they're at the document level only. Now do I specify two separate tables or one table with a flag? How will I render them? What would be a good UI for showing what parts are on which purchase order? Since EDI groups simply by presentation order, my plan of aggregating tags won't work at all - I'll have to write a line-by-line parser.

Man, this is gonna take 5...

You assertion that "hey, I've done something similar" would be like "Well, what the heck - I've moved elephants from NY to Seattle, and LA is the same direction and (I'm guessing) about the same distance, so I'll just throw on a simple padding and call it good" 

Then you get to the Grand Canyon. Oops, didn't have one of those on the way to Seattle...

[man, it's the analogy that just keeps on giving!]

Philo

Philo
Monday, July 14, 2003

But Steve, what if you do the $50k worth of research and *still* miss the big "gotcha"?

I sincerely feel you won't find every show-stopper until you've actually done the work, by which point you might as well actually do the work.

Philo

Philo
Monday, July 14, 2003

Philo, can you tell us your company name so I never accidentally hire you to do anything other than walk the other direction?

Can't beleive this clown
Monday, July 14, 2003

I'm saying Philo should've been truthful to his clients so they could've made the decision to either hire someone that knew what they were doing or else understand that they would be paying him to learn and experiment.

There are plenty of people that have written modem pool handlers out there, which is the hardest part.  Seems odd that the client demanded C#, but yeah that's not a big deal.

     
Monday, July 14, 2003

Philo, I'm going to venture a guess that you're a fairly new developer.

An experienced developer would've encountered SIMILAR issues in the other file formats.  There will almost ALWAYS be surprises, that's why they're called ESTIMATES. 

Your approach is similar to the developer saying it will only take him 1 day to handle file format X1 since he's already written a handler for format X0.

An experienced developer will remember that it took him 2 weeks to work out the kinks for X0 so will therefore budget AT LEAST that amount of time in figuring out X1.

And yes, there may come a time when your in week 1 of your two week schedule and realize that you've grossly understated your schedule.  That's why a schedule is NEVER etched in stone!

This really isn't rocket science.  You messed up, learn from it and make yourself a better consultant.

     
Monday, July 14, 2003

Philo, you argument is essentially:  I can't make a reliable estimate or keep a schedule, therefore all schedules are at best a guess.

I think it would do you well to actually read Joel's article that was posted above.

Jeff Cooper
Monday, July 14, 2003

To quote Dilbert:

PHB: "how long will it take to fix any bugs we might find in our program?"

Dilbert: "it is logically impossible to estimate the unknown".

Alyosha`
Monday, July 14, 2003

"Philo, can you tell us your company name so I never accidentally hire you to do anything other than walk the other direction?"

"This really isn't rocket science.  You messed up, learn from it and make yourself a better consultant. "

I find it quite interesting that "programmers" can so easily imply their ability to better estimate someone _else's_ schedule (given no details of the implementation).

Anyone with project experience knows that shit happens sometimes, period.  Missing a deadline given from estimate sucks, but it happens.  You can analyze the reasons for it the best you can, and try to avoid it next time - that's the point of this post.

Jeff MacDonald
Monday, July 14, 2003

Other Jeff - what the heck are you reading?

Philo's post was that an estimate is only a guess.  Other people's posts have pointed out that estimates may need revision, but they can be a lot more accurate than just a wild guess. 

Jeff Cooper
Monday, July 14, 2003

Not only is a schedule/estimate just a wild ass guess, specs are also highly overrated. 

Like I've said before, the code is the documentation/spec, and it will be done when it's done!


Monday, July 14, 2003

Jeff:
1) I've written modem services before, in VB
2) I've written over 100k lines of code in C# - code that's running in production applications

You don't feel that 1 + 2 gives a developer the ability to estimate a project (write a modem service in C#) accurately?

Philo

Philo
Monday, July 14, 2003

Philo,

Your analogies show that you basically understand how to estimate, but may not have applied the ideas on this particular project as well as you could have. To estimate accurately, you must really visualize the process of going from where you are to where you want to be. The difficulat part is to actually _investigate_ the cloudy areas rather than skip over them.

These are not the things that you need to pad your schedule for, because you can estimately them pretty well by applying a good amount of thought.

As far as the BSOD problem with terminal server, that is what you need padding for. There is nothing you could have done to predict this exact problem would happen, but there were risk factors that you could have seen from the onset, which could have helped you estimate the amount of padding required:

1) When interfacing with hardware, drivers can be a real pain.
2) You're using a relatively immature language and runtime environment designed for writing applications, not really for controlling hardware.

Without even knowing your requirements, I'd add at least one week of padding for each of these two issues.

Big B
Monday, July 14, 2003

You've written modem services before and you still made all of those mistakes you listed above? 

What's your definition of a 'modem service' anyway?  Have you written a modem pool handler before?

I would also like to know why the client specified C#?  What's wrong with the freely available code out there or the COTS packages?

     
Monday, July 14, 2003

Philo, et al....

I think, perhaps, my post was misread (or I'm just having a bad day and didn't communicate my thoughts well).

My initial post was a criticism of the two posts I had quoted above mine, not a criticism of the original post/poster.

Personally, I feel that we do our best to use our knowledge and experience to come up with an estimate that is true, like you said, at the time of conception.  Keeping this estimate accurate is dependent upon our change in knowlege and experience as the project progresses. 

Sometimes, however, we miss deadlines and it really sucks.  Worse yet is the fact that sometimes projects slip and when you look back there's nothing you would have done differently; you have no visibility as to why it slipped...it just "did".

Perhaps I didn't clearly separate the previous posters' comments from mine, but I felt them to be inappropriate and unconstructive.  My post was a poor attempt in defense of the original poster.  It would appear that the result of my post was the exact opposite of my intentions.

Sorry for the confusion.  Man, I should have stayed in bed today.

Jeff MacDonald
Monday, July 14, 2003

1) Yes. :)  The Terminal Services things really bit me - I worked over VNC before, didn't think TS was going to be the pain in the ass it was.
2) "Modem Service" - a Windows Service to handle incoming calls on four modems, authenticate the user, receive a file, log the transaction, and forward the file to a VAN. I wrote almost exactly the same thing in VB.
3) Why C#? The application it's integrated with is in C#, and the VB implementation is kludgy as hell (VB was never meant to be a service, or run multiple threads without blocking). I sincerely thought building it in C# would be *faster*, not slower. I didn't realize that nobody cared about modems any more. :)

Philo

Philo
Monday, July 14, 2003

What is up with all the anonymous cowardly flames?  There's no call for that (probably all the work of one or two people anyway).


Anyway, sometimes you just get bit in the ass and something ends up taking 3 times as long as you estimated, even though you know *exactly* what needs to be done.  However, you end up wasting a week because of a bug in an updated library, or the API doesn't work as advertised, or whatever.  These are the most frustrating causes for schedule overrun.

Crimson
Monday, July 14, 2003

3X the original estimate is classified as 's*t happens'?
Even when the writer/estimator had done something nearly identical before?

I've got to think maybe philo is posting as other users.  I don't see how that is defendable at all. 

Do it twice where I work and your looking for new employment.

Muayyad
Monday, July 14, 2003

"What is up with all the anonymous cowardly flames?  There's no call for that (probably all the work of one or two people anyway)."

I second that!!!  If you have NEVER had a project go past the deadline you're either:
1) Not a programmer
2) Never written anything for production.

The amount of worthless flaming in this topic has just lowered my appreciation for JOS readers a notch.

shiggins
Monday, July 14, 2003

OK maybe that was some worthless venting also.

Philo,

What I have done in the past is ALWAYS pad my estimates to accommodate for "sh*t happens".  However, I don't believe that is worthless buffering.  On projects where I came in on schedule, I used that time for rigorous testing and/or adding cool features. 

However, I agree that recognizing that estimates are best guesses, gives me an uneasy feeling.  I believe that's because of the nature of programming.  In development perfection is king.  However, you CAN'T have perfection in estimating. 

shiggins
Monday, July 14, 2003

I work for a IT consultant with budgets and sales forecasts.  We do some fixed bids, but most of it is charged by hour.  If you routinely slip, then you are costing more money than you generate.  Maybe this is why so few companies are able to be very successful in IT consulting.  Or why the ones that do succeed do so despite of themselves.

Muayyad
Monday, July 14, 2003

Wow, what a heated topic!

One thing I noticed that seemed to be missing from the subject of estimation and accuracy is the subject of granularity.

In my experience, the delta between the time required to complete a task and the time estimated varies wildly both under and above the estimate.

(For a full explanation of why tasks are almost never completed early, see "Critical Chain" by Eliyahu Goldratt, or download this presentation:

http://www.braithwaite-lee.com/opinions/agile-critical-chain.ppt
)

When there's only one, indivisible task involved, I would be very surprised if I could reliably estimate it. The trick, as Joel  suggests, is to break the task into finer grained tasks and estimate those. Now statistics come to the rescue and the underages balance the overages and you hit the total.

Only they don't actually balance if you divide the task and move forward blindly. You have to divide the time into a finer granularity and establish regular iterations or milestones where you adjust priorities and manage expectations. You need a feedback loop where you use what you've learned.

(this point was made earlier by the poster who used XP spikes as an example).

Everything I've leanred about managing projects, be it Agile or Classical flavoured, consists of ways to dividing a task into finer grained sub-tasks and milestones in order to deal with the very high deviation from the estimate for each task.

So...

Philo, if you ask me to move Elephants from New York to Los Angeles, my instinct is to never provide an estimate off hand, even if I've moved Elephants before. I need to break things down into tasks and set up regular reviews.

See you in Kansas City!

--
http://www.braithwaite-lee.com/

Reginald Braithwaite-Lee
Monday, July 14, 2003

"3X the original estimate is classified as 's*t happens'?
Even when the writer/estimator had done something nearly identical before?

I've got to think maybe philo is posting as other users.  I don't see how that is defendable at all. 

Do it twice where I work and your looking for new employment."

Easy enough to deal with that:
"I need a routine to read a string and split it on commas, returning a string array"
"Good enough. Three months."
"Three months?!?"
"You blow your lid when I run over my estimates, so I'll make estimates I know I won't overrun. Oddly enough, you don't seem to throw money my way when I turn in work early..."

Punish people for optimistic estimates and the padding will get larger every time. Except that, as Joel has pointed out (and I agree with) projects tend to expand to occupy the allotted time, so this is a losing proposition for you.

I also notice that nobody asked if I notified the client when I was delayed (I did), or who this delay hurt (it's FFP - you do the math). Finally, will you all calm down if I tell you I just finished a project budgeted at six weeks in three? Or will the "exact estimate" crowd still be annoyed that I didn't take three weeks to properly spec the work first and turn in an accurate estimate?

And for the [ahem] person who accused me of creating sock puppets - bite me. :-)

Philo

Philo
Monday, July 14, 2003

a bad estimate is a bad estimate, no matter who's pocket book it eventually lines.


Monday, July 14, 2003

This is a question from ignorance, but shouldn't using C#, which appears to resemble Java or VB in terms of levels of abstractions, to do a project involving interfacing with hardware, make you worry about unexpected snafus?

Stephen Jones
Monday, July 14, 2003

As far as making estimates goes, I follow the Scotty approach and double whatever my best guess is. If I finish early, people are happy.

I know of no reliable ways to make estimates. Something unexpected *always* comes up, usually in the form of someone asking for new features that are slightly tricky to implement.

And then there are things like...

The author of a library you depend on can't answer your questions quickly, because he's asleep, in France.

Customers that purposely call late because they know "the programmer" might accidentally pick up.

Fred2000
Monday, July 14, 2003

One thing I would suggest is to take a longer term view of your estimation success rate. If most of the estimates are way off, then there's a problem. If almost all are very close, with only a small number way off, then it's probably just bad luck. Despite what some posters have said, nobody hits the bulls eye every time.

So Philo, is this just an aberation from the normally successful estimation or a frequently repeating problem?

Bill Tomlinson
Monday, July 14, 2003

In the past it's been a problem due to my overoptimism. Many beatings have taught me to temper my enthusiasm, and I'm getting much better, down to the occasional trip and fall... :-)

Philo

Philo
Monday, July 14, 2003

"Do people really prefer the $125k cost in (a) to the $75k cost in (b)?"

No they don't, not in the UK, not unless they're a government or similar institution that habitually reserves separate budgets for feasibility.

You're average company (from family firm to 50 million sterling turnover), isn't interested in paying the proper price for a feasibility study or even modelling what they have now.

They're solely interested in the final product, so in fixed price terms (which are now more the norm and which I've always tried to do), you have to weigh up the possibility of getting ahead on the deal or the resources in order to properly do the planning.

If you don't and you quote up front for a complete investigation, feasibility, requirements spec etc,etc you will lose out to someone else.

Simon Lucy
Monday, July 14, 2003

More than a completeness issue, your dilemma sounds more close to the uncertainty principle.

Software is “universal”; some problems may well be irreducible, meaning that the only way to get there (finishing your product) is by getting there with no shortcuts or higher level abstractions. Lego blocks and guesses are allowed though, but the truth shall be revealed only when development is finished.

Pablo
Tuesday, July 15, 2003

You people are pathetic!

This is yet another reason why jobs are going overseas.  You get developers, on average that are more educated for less money.  And they write good code and it gets done when they say it will get done.  Something like 85% of the CMM level 5 orgs are overseas, and the majority of level 4 orgs are overseas too.

wake up & smell the outsourcing
Tuesday, July 15, 2003

You'll have to explain to me sometime how the laws of physics work differently in those other countries. Or are you simply saying they're better at estimating?

Philo

Philo
Tuesday, July 15, 2003


I can't remember where, but there's an article by Joel in which he says that third-party components (if .Net libraries can be called this way) will put you in trouble as much as they will help you. At least if you have the source code, you can fix it.

Another thing I noticed : the tasks generally take me three times more than expected - or three times less.
But obviously, if two tasks are expected to take three hours, and one takes 9, and the other 1, then that makes 10 hours - and you're late.

GP
Tuesday, July 15, 2003

CMM doesn't measure how smart your programmers are.  It measures how entrenched your software process is and how risk-adverse your company is.  If all you do is clones of previous projects, chances are you rate much higher on the CMM scale than a truly innovative company.

As DeMarco and Lister said, "Organizations everywhere are under pressure to climb the CMM.  In the extreme, they are hell-bent for Current Level+1 by tomorrow, or else.  This is the Dark Side, for it entices the play-it-safe behavior of low-risk, and therefore low-benefit, projects."

Alyosha`
Tuesday, July 15, 2003

Come on people, you're not working on anything that the CMM doesn't apply to.  There's a handful of programmers that are working on anything that's remotely bleeding edge.  Hint:  Writing .NET code ain't it.

     
Wednesday, July 16, 2003

Doesn't CMM apply to *you* repeating a process? (As opposed to doing something that's been done before somewhere on earth). IOW, "how long did it take us to do this last time?"

Does anyone know if the NASA shuttle folks can schedule their changes accurately? They're CMM5 - when they get a directive to deal with a new orbit do they say "that'll take nine months" and they're done 270 days later?

Philo

Philo
Wednesday, July 16, 2003

If by "high-tech" you mean projects that are earth shaking, revolutionary ideas that will change life as we all know it ... okay, yes, very few programmers will ever work on something like that.

That's not to say that all projects are equally difficult.  Some are more complex than others.  All are unique in at least one way.  And, like Philo said, CMM doesn't deal with what's unique and innovative in the world.  It deals with projects that are unique and innovative within your organization.

Like I said -- you can only be a CMM level 5 if all you specialize in is drudge work that can be done by any trained monkey.

Alyosha`
Wednesday, July 16, 2003

I guess the NASA JPL programmers are just code monkeys then.  They have something ungodly like a 99% accuracy rate in delivering software according to estimate.  I forget where I read that (maybe McConnell) and I'm not going to bother looking it up because it wouldn't do you guys any good.  Continue on your path, I will own you one day! :)

you clowns are killing me
Wednesday, July 16, 2003

Sorry, NASA SEL, not JPL.

you clowns are killing me
Wednesday, July 16, 2003

Dear "You Clowns" - enlighten us. Regale us with tales of projects you've worked on that were delivered on time and on budget, and where that budget was won competitively.

Philo

Philo
Wednesday, July 16, 2003

To all of you estimating experts, I still don't see how to account for bugs in the platform (OS, runtime, 3rd party components, etc) in your estimate.

Other than tossing in an arbitrary multiplier based on experiance what do you do?

If you have sufficient historical data (e.g. you are working on the 10th revision of a reasonably stable product) it might help, but it sounds like in this case, it's a one off.

pdq
Thursday, July 17, 2003

*  Recent Topics

*  Fog Creek Home