The Old Joel on Software Forum: Part 3 (of 5)

Software Craftmanship

Just as an aside - with your particular problem, I wouldn't spawn a subprocess at all.

The user will not be performing task X more than once in a blue moon, so waiting a minute while it's carried out isn't a massive problem. Obviously, feedback is necessary, but a separate thread strikes me as unnecessary. I'd simply have a progress bar that told them what was going on, but not allow them to take any action during that time. Not quite as nice for the user, that 0.01% of the time that it happens, but a heck of a lot faster to put in, and I'd have thought that the time could be better spent elsewhere in the system, improving something that happens more often.

Andrew Ducker
Tuesday, December 2, 2003

I prefer a more pragmatic approach. If it is rare to include 120MByte files, why not ask confirmation for Really Big files (configurable). This way the user knows it is going to take a while and has a chance NOT to do it if it's a mistake.

If the user may hit cancel during the operation, it is MUCH more work (and chance of bugs) to undo a partial operation.

craftmanship is also about deciding that 2 mm off can be tolerated or masked by using a different strip. Unless you want to build a really expensive house where every screw is lined up....

Adriaan van den Brand
Tuesday, December 2, 2003

I certainly agree that aspects of software engineering require craftsmanlike qualities, but that is only one part of the toolkit which can be deployed when appropriate. Personally, I tend to see software engineering as something like medieval stonemasonry. One day, we will have the tools, history and momentum to pull virtually everything we need to build a system off the shelf, including all of the infrastructure, plumbing, etc. but for now we have to make almost every peice by hand.

Modern manufacturing doesn't tend to sit people in straight lines doing repetitive work. It tends to organise people into workgroups, allow those groups to design their own work up to a point. They also have plenty of infrastructure to give control and visibility of their work inside and outside the group. Some teams are homogenous with everyone doing similar things, other teams are more structured with a highly skilled individual doing a specialised job supported by others (acolytes & apprentices ?). That skilled person could be a craftsman.

This isn't a million miles from a modern software team. I have had teams organised around a 'star' and I have had teams organised to churn out large volumes of homogenous code. In the latter case, it felt like a production line, but at both extremes the quality has been excellent.

Ian Sanders
Tuesday, December 2, 2003

I'd just like to point out that printing presents almost the same problem as this "reading in a really big file". The MFCs have a solution in there for that (okay, it is really really ugly, but it is a solution). Why not just rework that? Actually it strikes me that this is a problem for which there should be a good design pattern solution, saving craftsman-time.

Tuesday, December 2, 2003

I was wondering which process brought this item to the top of the "What can I do today that will net the most increase in sales for version 3" list.

Just me (Sir to you)
Tuesday, December 2, 2003

"I was wondering which process brought this item to the top of the "What can I do today that will net the most increase in sales for version 3" list."

Probably a different process from the one that encourages a fool like yourself to whine about someone who goes to the trouble of offering a resource such as this one to actually have the temerity to write about their development activities.

Tuesday the 2nd
Tuesday, December 2, 2003

I have to agree with Sir here.

Frederik Slijkerman
Tuesday, December 2, 2003

You can call it whining if you like. I was just surprized. The image I have of FCS is of a company that presents a well focused and tuned development process. Joel has one of the best public advice available to improve your process.

I feel tactical prioritization of development effort is part of a great setup. There is always more ideas and improvements possible than there are recources to spend. Every one of those ideas and improvements has the potential to delight the user, so though choices must be made every day.

The FCS prioritization process, wheter that is a formal system (managed through FogBugz?) or something implicit, threw up "GUI feedback for large file import" as the top item for the day. To me, as a complete outsider and obviously not a well organized person (why else would I be posting on JoS?), this is surprizing.

My remark is not about wether adding GUI feedback for long operations to CityDesk is a good thing, nor is it about how to achieve this. It is about the FCS development process, which I believe is a reasonable topic for this board.

Just me (Sir to you)
Tuesday, December 2, 2003

I have an instance of Joel's problem in my code. I didn't create a thread or child process to solve it. What I did was establish two threshhold values that say anything (number of calculations, size of file etc) over X takes a long time on a slow/fast machine. I use GetSystemMetrics to determine if Windows thinks the machine is slow. If the machine is slow and we are above the slow threshhold value or if the machine is fast and we are above the fast threshhold value then I pop up a (can be MDIChild or not) form that only contains a progress bar. The routines that comprise the I/O and calculations are broken into small enough steps that adding a DoEvents (or two) to the main loop is enough to allow the user to continue doing other work while the operation is in progress. I haven't been able to bring this peice of code to it's knees either, though I don't have a 90MHZ Pentium to test it on.

Dave B.
Tuesday, December 2, 2003

I would be inclined to think this was driven by some user feedback. I have a difficult time imagining Joel sitting at his desk one day and accidentally dragging a 120 MB HTML file in to CityDesk just to see what happens.

Perhaps some user somewhere was extremely concerned about large file imports. Or more likely, I think, is that Joel is running on a very fast machine compared to some users who are importing relatively large files (10+ MB), and the equivalent experience occurs for them. Certainly it may be that a user on a P II 800MHz with 32 MB of RAM importing a 10MB file (relatively reasonable although still likely excessive) is approximately the same experience as Joel importing a 120MB file on a P4 3 GHz with 2 GB of RAM.

I'd like to hear Joel's reasoning for this, not to give away secrets or ruin any competitive adavantage, but rather to see why it was that he tried to import a 120 MB file in the first place. That's certainly an edge case for a program such as CityDesk.

Lou
Tuesday, December 2, 2003

Ian - I would like to understand your comments a little better.

Stone masonry, plumbing and carpentry are not "off the shelf" type actions. The materials, bricks, pipes, lumber, may be easier, but the task of assembly is still a craft. Consider a bookshelf. The one built by a craftsman will look far different than the one built by me. Yet, we both have access to the same tools, and materials. I would even gamble that two craftsmen would produce different end products.

In modern manufacturing, outside the idealized situations we see in Tom Peter's books, the work is done by humans because it is cheaper. It inexpensive automation could accomplish the task it would. You took this a step further with "I have had teams organized to churn out large volumes of homogenous code."

It would seem homogenous code would be developed once and reused (hence the homogenous). Or, did you mean the simpler work, whereby the complex code is left to the masters?

MSHack
Tuesday, December 2, 2003

I can't help but feel that those of you who are wondering what process led to Joel deciding that this fault had maximum priority and must be done really did miss the point.

The piece is entitle 'Craftmanship,' not 'Engineering.'

Take a look at an old Cathedrel. Europe has lots of them.

The Gargoyles do not make the spouts any more effective at the conveying water away from the building. They do, however, look stunning. The Cathedral is the result of Craftsmanship.

Take a look at a modern structure. Does it have the same character as the old cathedral? Sadly not. These buildings are the results of Engineering.

It seems to me that Joel is talking about making that extra effort to make a product wonderful, not just functional.

You can't do this just by reacting to external forces, it takes personal vision.

Ged Byrne
Tuesday, December 2, 2003

Ged, I'm more interested in hearing why that particular feature was chosen as the next to be worked on. I'm having a horribly difficult time imagining a 120 MB file being part of a CityDesk installation, it seems quite out of the context of the application.

It is as if you were the designer of the Cathedral and you noticed that someone might come to mass with 120 members of his extended family, so you decide to spend three months reworking the seating system so all 120 of them can sit comfortably next to each other rather than having to scatter throughout the Cathedral.

Certainly that user might apprecite the convenience, but everyone else wouldn't care and might wonder why the organ hadn't been tuned yet.

Lou
Tuesday, December 2, 2003

I believe what Joel's doing just makes sense. It's something I do all the time. I test the extremes of my code. Does it work with small, medium and large size inputs and outputs. Does it work with 0, 1 or many inputs/outputs. Does it work with this kind or input, that kind of input. In doing so you define the limits of your application. I do this with most code that I write. Maybe it's because I'm too much of a craftsman and attention to detail has gotten the better of me. I'd prolly do well to have an engineer around to tell me to stop nitpicking.

Dave B.
Tuesday, December 2, 2003

What would make even more sense is if CityDesk kept a log of metrics of your various methods of publishing - FTP, file copy, etc. and their speeds.

Then when you dropped that 120mb file into CityDesk it said:

"Hold on there buddy, you do realize that you're on DSL and this file upload will take all night, and because of a quirk in CityDesk, we'll want to upload it each and every time you make a minor change to your site."

www.MarkTAW.com
Tuesday, December 2, 2003

Usually Joel makes the most sense of anyone I read and it's like watching a Robert DeNiro or a Samuel L.Jackson movie to me. I don't care what role they do - it's always a delight! Joel's writing is that way.

Usually!

Not this one though. I think Design is "Appropriate Engineering". Doing a lot of additional stuff for 120 MB files is a lot of overkill. No matter what your product is, it's always designed with some limits in mind. What if the file is a couple of terabytes. What happens then?

Design is like ensuring uptime. Going from 95% availability to 99% availability costs soar. You can always design for whatever performance levels you want but craftsmanship is also choosing the right level of design for the limits you can be explicit about!

Just my 2 cents!

Nari Kannan
Tuesday, December 2, 2003

I agree with you Nari, but I also think that by Version 3.0 most of the limits of a product should be established. I think Joel found another limit of his software that he feels deserves attention. The fact that CityDesk appears to lock-up (when it's really just busy with no visual indication) is enough to warrant the new code. I would give the same attention to my software.

Dave B.
Tuesday, December 2, 2003

The 120MB file thing is (quite) strange. But it's only an (exaggerated) illustration of the point Joel was making.

Because he was not really talking about 120MB uploads, it did not make much sense going into the painful details of why the change was made. That is, one could assume, at least for the "craftsmanship" discussion, that the 120MB work made sense.

Some people clearly misunderstand the following: not _providing_ the rational does not mean that there is no rational.

Anyway, the 120MB test on a fast connection might be a very good test of a much smaller file on a slower connection.

njkayaker
Tuesday, December 2, 2003

Let's have a few things cleared up:

Some of you seem to think this is a "strive for excellence, do the right thing" vs. "let's do a quicky and pull the wool over their eyes just long enough to grab their money" issue.
It is not.
It is about choices in excellence.
Excellence doesn't come for free. It requires recources, mostly in terms of time and effort. These recources are limited. Effort spend on improving feature X automatically means that effort is not spend on improving feature Y.
Suppose you don't even care about sales, just about product quality. In an ideal situation you could lay out all of your components, and know how much a given unit of "effort" spend on each one would increase the overall quality of the product. Optimal effort investment then simply becomes working on the component that shows the higest marginal product improvement.
In real life we do not have the luxury of complete knowledge (and we have several extra constraints besides quality alone). Still, all else equal, a company that has a better task selection or priority management system will climb faster towards an overall better product, since at each point in time they choose better roads up the mountain.

Optimal task selection is not obvious. We are distracted all the time by the current thing we are working on. We like doing things not only good, but better. This leads in the extreme to things like shaving the last few microseconds of code that will only execute once etc. But doing better on that particular item might mean the overall product suffers, since you might have to do a rush job on something else later, or drop certain things from a release all together.

How do you manage task prioritization? Which products are used to support this? Do you try to measure and improve this process?

Just me (Sir to you)
Wednesday, December 3, 2003

MSHack - replies below.

With respect to stonemasonry (etc) I meant this as an example of a craft which used to require much more skill than is needed today and is a metaphor for today's developers. Every block of stone used to be carved by hand, even the plain ones. Now they are made by machines. Putting them together becomes a design & engineering problem. Using many identical components means that you can characterise the behaviour and quality of those components very reliably. That means less waste. Not all bricklayers are craftsmen, some undoubtedly are. If it was possible to automate the bricklaying process, someone would have done it. The cost and engineering issues of working with a vertical work surface in the open air in a variety of environments mean that it can't happen yet. I'm sure most architects would love a robot bricklayer because, again, the characteristics of the walls that they would build would be much more precisely understood than those built by people, so they could be bigger and so on. You can argue that cathederals are massively overengineered partly because of the need for large numbers of individually crafted components with poorly understood properties. I think that software engineering is at that stage today. The emergence of patterns, open source and third-party components is helping us move on, and not a moment too soon. I like to work with software craftsmen, but I also like to have a few plumbers and bricklayers around who are learning to be stonemasons and architects.

When I refer to homogenous code I mean large amounts of similar code, not identical code. Obviously, I don't have large numbers of people writing identical code over and over - that is abstracted out and re-used. In that kind of environment, most of what is left are the simpler problems.

Ian Sanders
Wednesday, December 3, 2003

Craftmanship is among other things choosing the right tool for the job.

It is not surprising that there is problems with creating multitasking applications in C/C++. A craftsman would choose an more apropriate programming language.

Large system and/or multitasking means that you should be using Ada. Or Eiffel. I don't know if Delphi is sufficiently robust with regards to multitasking.

On the other hand, not everybody is into reliable software.

greetings,

Tarjei T. Jensen
Thursday, December 4, 2003