Fog Creek Software
Discussion Board

Theories on why programming is hard

I agree with Joel.  Shipping an app that meets user expectations has only gotten harder.  I would argue this is for (at least) six reasons:

- Performance optimization.  Why 3-tier?  Why not a client app that uses an Access database in a shared directory over a WAN link?  Two of the tiers are basically performance optimizations, albeit absolutely necessary.  Each tier has it's own language and technology (MFC, DHTML, ASP, SQL, etc.) and often separate developers.  In many cases, much of a 3-tier app's complexity exists to solve what amounts to a speed issue.  So, you have three groups that don't speak the same language and are thinking about a business problem in three different ways.  Can you be a _good_ SQL Server DBA, a _good_ DHTML layout person, a _good_ middle-tier guy, have a _good_ understanding of the business problem, and be a _good_ project manager?  Not really and still get anything done.  The more technologies force us to specialize, the more frictional waste you'll get by fragmented teams.

- Security optimization.  This feeds the necessity for 3-tier.  I would argue that we're not that many years away from solving these issues.  Smartcard authentication, "skip a tier" delegation, and universal wire/memory encryption will help.  The whole "we need a middle-tier to keep people out of the database" argument is weak for many apps.

- Non-linear code.  Events and threads rule now.  Remember UI functions like:
  void RunInvoicesTask();
  void EditLine(int lineNumber);
You were guaranteed nothing else would happen in the UI while the task was open.  When editing a line, you polled for keystrokes.  Down arrow?  Write the record and return.  Those were the days.  I'd argue GUIs have been a "good thing" in almost all instances, but they've increased the "what could the user possibly do at this instant in time" possibilities by at least an order of magnitude.

- Lack of continuity.  I don't understand how shops where developers only stick around for 18 months get anything done.  Constant switching of methodologies, technologies, and programming standards leads to "hard" development.  In my mind, the only way to avoid this is team continuity, developer understanding of the end-user, and developers having to live in the barn they helped build.

- Technology for its own sake.  XML, data access abstraction layers, COM+, insert/update/delete stored procedures, threading, COM wrappers, etc.  You can write an app that uses all of these things, and afterward, they all seem necessary and matter of course.  I see a lot of projects, though, that are simply made harder than they need to be.  How to combat this?  A project manager that understands "why" in addition to "how".  There's a theory that cost goes up by the square of complexity.  Make sure you're getting your money's worth.

- Proficiency.  What is an acceptable level of proficiency?  Low proficiency means incorrect or unstable code gets written, lived with, and worked around.  This is hard.  One easy technique I use to rate my own proficiency is to make a ratio of the number of newsgroup postings I can answer.  10%, not proficient.  40%, getting there.  80%, arrived.  I thought I knew SQL Server until I took this little test.

This is what I can come up with.  Any others?

Bill Carlson
Thursday, December 12, 2002

- Boredom. Most necessary programming tasks (fix 200 bugs left by the guy who just quit) are not interesting. Most software projects that are useful (drug procurement system for a hospital, Microsoft Excel 2010) are by no stretch of the imagination,interesting. Thus, if you are like most programmers, and are in a position like this, almost everything else in life (getting a soda, reading joel on software, taking a dump) is more interesting than the task at hand. It is very hard to focus on getting any work done.

In order to get boring software tasks done, you have to first convince a bunch of malleable (i.e. fresh out of college) people that they are Really Smart, and Special. Then you have to convince them that they will ultimately end up with a lot of money if they spend 5 years of their life working on boring shit, right now. The reason Microsoft's culture is the way it is, is because it HAS to be that way, or else all the smart people there would leave, and the Norwegian spellchecker in Word would never get finished.

Thursday, December 12, 2002

Programming is hard because _everything_ in life is hard.  Even those people who are born wealthy have it hard, though for different reasons than the rest of us.

J. D. Trollinger
Thursday, December 12, 2002

McDonald has a point, but only in software would "entertaining for the engineers" be a consideration.  Imagine a building with wavy floors, useless spires, unsupported staircases, horizontal elevators, etc.  Getting someone to sign off on it because it would "entertain their engineers" isn't going to work.

Personally, I think a lot of punk-ass programmers need to go back to business school.  (just an observation; not talking about anyone here)

I agree wholeheartedly that most development work is boring.  However, I can't condone screwing up a project for entertainment purposes.

In other industries, a conclusion has been reached that the human mind is only capable of X consecutive hours of an activity.  Tech support often is only on phones 5 hours a day, etc.  Some of us would gladly stuff envelopes for two hours a day rather than sit at our desks, surf and pretend to be working.  The human organism has limits and perhaps enterprise development is taxing those limits.

Anyway, good point...

Bill Carlson
Thursday, December 12, 2002

I disagree with Trollinger.  How do you tell if something is "hard"?  Well, one way is to look at variations between teams.  In architecture, the cost of designing a building may vary 30% from firm to firm.  In software, the variation can be 1000%.

There are lots of smart people in software development and they don't agree on squat.  Is it because we're disagreeable?  Yeah, but it's also due to the subjects being poorly understood and, well, hard.

I'd also argue teaching is hard due to similar variations in performance between teachers of equal motivation.

Equal motivation is the key.  It's not hard to smell nice, even though some people don't do it.

What better definition of "hard" than if bright, motivated people attempt something often and are often not successful.

Bill Carlson
Thursday, December 12, 2002

Imagine a talented architect designing a new building. Cool.

Now imagine the architect sitting at a CAD workstation detailng the electrical and plumbing systems for 3 months. Bo-ring.

Your mission is life, should you chose to accept it, is to maximize the cool and minimize the boring.

In some ways each is "harder" than the other.

fool for python
Thursday, December 12, 2002

"What better definition of "hard" than if bright, motivated people attempt something often and are often not successful."

Some computer science prof (software pundit?) made the observation: "In science, progress is made only because we are standing on the shoulders of giants, in software, however, we step on each other's feet."

Thursday, December 12, 2002

I must be a nut, I enjoy the mundane, business rules and such.

Ryan Ware
Thursday, December 12, 2002

Other reasons exist:

UML/Other tools, i.e Rational etc - a developer needs to know this in order to develop?
In some cases this is even more difficult then the technologies to build the project itself. It's like, build a space rocket using some tool, and then use the space rocket to develop the simple accounting application.

On the boredom side - Testing, we all think it's great, right?
No, wrong, we all hate it, especially unit testing, it's dull boring uninteresting work, and it rightfully fills us with loathing. We only say we like it because its politically correct to do so. Throw in more software test tools and the knowledge you require to write the application just increased(again).

Business Analysis, mosts corporate BA's are useless, if only the skill of BA's was the same as most developers, then the project might get off on the right foot. The BA is a manager wannabe.

Politics - You've busted your gut developing something and just found out you need to be a lawyer to get it deployed. You have no access rights to any deployment directories, it's going to take days/weeks from here to deploy and test in the production environment, all you need someone to do is to create a directory and give you access to it. The Sys Admin is a prima donna, he's talking security, yada, yada.
You give him 7 alternative strategys all of which guarantee no security risk, he can't get it done until next week.

Large Companies - Before you even start your development, you've gotta present the application to a company architecture group for approval, they have a standard application document, none of the questions are relevant to the application you are working on.
Also, see politics above.

I could go on....

Thursday, December 12, 2002

Oh, and some permanent employee on the team's just been to a microsoft conference on MTS (or something) and suddenly that technology is central to the development
of your simple accounting application.

Thursday, December 12, 2002

Programming (i.e. software development) is hard because of all of the reasons provided so far by previous posters.  However, it is also hard because roles and responsibilties are not well established in this industry.

Some programmers must wear all the hats, others must wear many hats, and then there are some who only have to worry about banging out code all day long. 

DBAs, Business Analysts, Project Managers, UI experts, etc. -- these specialized IT workers simply don't exists in many employment situations and when they don't exist whom do you think gets stuck doing those tasks?

I have yet to work in an IT enviroment that was similiar to one I have worked in before.  When I change jobs, I find that just about everything tends to be different and in many ways I find myself starting from scratch!

As poster McDonald mentioned (I have changed his words) "in the IT industry (commercial and non-commercial) we step on each other's feet instead of standing on the shoulders of giants".  Software engineering practices and computer science concepts simply aren't taken seriously by most employers.  Also, many employers (unfortunately, I have to agree with this claim) claim they simply cannot afford to take them seriously.

one programmer's opinion
Thursday, December 12, 2002

Programming is sometimes "hard", and most other times just boring and mundane as you re-learn and try to keep up with the ever growing API's and tools which abstract so much and cludge together rather than involve precise engineering and efficiency.

Your simple accounting package could probably be created using pure Win32 and spit out a 100k executable.  But then someone comes along proclaiming "methodologies" and loose statistics about how much more "results oriented" this new approach will be and how this R.A.D. tool de jour will eventually replace good programmers.

Programmers need jobs too.  And so now your "simple" accounting package has blown up to require distributed processing, and has dependacies on other technologies which change faster than most people change their underwear.  Nevermind the maintenance of such a beast.  Well now I guess network software administrators need jobs as well.

And the cycle continues because time is money, and if you can do it the "easy" way, then why choose the "hard" way.
So is life in software development within the internet age.

I'm personally into embedded development and so nothing really comes easy from that low level.
But there still are some boring times, let me tell ya.

Sebastian Dwornik
Thursday, December 12, 2002

I don't think it is the problem space that has gotten harder. We are still solving same problems that have been around for 30 - 40 years. Its the solution space that has exploded. To do the job, you need to know alot, have done alot, and are still doing alot.

The days of Ivory Tower Architects are over, way over. These architects don't understand the ground level, where code meets electrons. You just can't do a good job, if you are not coding and learning all the time. You can't just stop learning. This goes for developers as well. If you don't stay up you become obsolete.

I would almost have to say that Moore's Law applies to software as well. The amount of API code out in world double every 18 months.

Dave Hickerson
Friday, December 13, 2002

I'd tend to disagree with the previous post. In my experience, those developers who are concentrating on "where code meets electrons" are too busy trying to keep up to actually design anything.

We, as a community, have a lot of these problems because we don't learn from our past mistakes. In general, we still don't really listen to what the customer wants.

Did you notice that, in the original posters list of issues, very few, if any, actually make any difference to customers?

Ask a fellow programmer what their favorite project was. Then ask them if it was a commercial success. Odds are it wasn't. Odds are the programmer doesn't care. With that to contend with, why are we surprised that programming is hard? We make it hard.

Bruce Rennie
Friday, December 13, 2002

Bruce - just a comment:

We, as a community, have a lot of these problems because we don't learn from our past mistakes. In general, we still don't really listen to what the customer wants.

Perhaps indirectly you've hit the nail on the head.  The problem isn't necessarily that we don't listen to what the customer wants.  I think it's a consequence of our languages, which are unable to define exactly what we mean.  Everyone has different experiences.  Therefore it only makes sense that they have different interpretations on what X or Y mean. 

Customers IMO have no good/specific way of telling us what they want.  And few clients want to pay for the X number of days that an architect/developer should just sit and watch a user doing his/her job related to the system to be developed. 

Friday, December 13, 2002

"We are still solving same problems that have been around for 30 - 40 years."

I can't agree here.  30 to 40 years ago, we didn't even have C (which was first created 30 years ago, in 1971-1972).  We had completely different problems.

Even 20 years ago (around 1980), programmers were trying to figure out the best way to draw a line of pixels.  They were writing their own versions of strlen().  State-of-the-art game development was the text-only STARTREK, where *jumping to another location in the code* had intricacies and was treated with some delicacy.

We're not solving the same problems that we were in the past; we're just solving a different set of problems, which are at least as hard.

Brent P. Newhall
Friday, December 13, 2002

"We are still solving same problems that have been around for 30 - 40 years."

"I can't agree here.  30 to 40 years ago, we didn't even have C (which was first created 30 years ago, in 1971-1972).  We had completely different problems."


I have to agree with some of what Dave Hickerson wrote.  We are still solving some of the same problems that have been around for 30 - 40 years and the solution space has exploded (i.e. some of the examples you used in your post).

I don't know about you, but several year ago, I was required to maintain several mainframe legacy software systems (structural), PC applications (procedural), and client/server systems (object-oriented) all at the same time!

So, I have to say that many people still are solving the same problems that existed in the past, while at the same time, dealing with many new problems.

one programmer's opinion
Friday, December 13, 2002

Programming is like join the dots, well not that easy, but pretty much that in a nutshell.

One has to join the dots between the problem(whatever your domain is finacial/ energy/ software) and the solution(language of choice).

Not understanding the problem completly, and not having through knowledge of the language are the main reasons.

Prakash S
Saturday, December 14, 2002

*  Recent Topics

*  Fog Creek Home