Fog Creek Software
Discussion Board




Specialization of developers good for a business?

I'm looking for resources (anecdotes, ideas, and mostly books/articles/studies) about what effect the specialization of developers has on an organization.  Should I try to make everybody know everything, or aim for "focused on one thing and interested in others" or is "totally focused and excellent in one thing" best somehow?  What about short term and long term considerations? 

Nathan Arthur
Friday, March 07, 2003

XP tries to avoid specilization, as does the general Microsoft hiring practices.

Most of the arguments for either way are pretty much common sense.  If you hire a graphics maven and then don't end up needing a graphics maven, you either end up with somebody who is unhappy, laid off, or who leaves the company.  If your graphic maven leaves the company or is hit by a truck, you are screwed.

On the other hand, forcing somebody who just doesn't have the abstract stuff in their head to understand graphics to write a 3D engine is also asking for trouble because the graphics maven could get it done in a fraction of the time and write something that is much much better.

A comprimize is prolly better.

flamebait sr.
Friday, March 07, 2003

You need both specialists and generalists, with the majority being generalists.  The specialists can lay the architectural foundations and give direction because they know the ins and outs of the particular platform and language. While the generalists (if they are generally smart!) will be able to pick up enough to do the job, as long as the specialist is available for answering their questions and reviewing their code.

T. Norman
Friday, March 07, 2003

The choice of specialist versus generalist actually has a logical solution.  Each system has a throughput limiting activity - the constraint.  In order to elevate your constraint, you should get someone that specializes in the constraint area.  You don't want this guy doing anything else because every hour he gives to something else is an hour of throughput lost to the whole system.

In order to further elevate this constraint you can try to hire generalists in other positions.  Since these folks have excess capacity, it makes sense to try to get them to use this excess to help out your constraint resource.  So, if your constraint is testing, maybe you hire some programmers with some test-friendly abilities that can do some of the simpler stuff.  Cross training against your capacity constraint is always a good idea if it is possible.  Steve McConnell has some great material on how to properly do this on his website at http://www.construx.com

Because your system is so sensitive to the performance of this constraint, you need to make sure you have options available if something happens to it.  If the guy quits, you need to have a plan to immediately farm out the work to a contractor or something similar.  You also need to make sure this guy never runs out of work, so keep some ready for him at all times.

You might also want to look at what are called "near constraints" These are things that are almost capacity constrained or that will become the next constraint if the current one gets elevated.  It pays to plan ahead.

The other thing you need to consider when staffing up is quality.  If you put bad material through the constraint that causes rework, that is output lost to the system.  You should inspect each input to the constraint to ensure quality is as high as possible.  That might motivate you to hire specialists upstream as a hedge.

pair designer
Friday, March 07, 2003

I've been working on a theory that says "Never have a specialist who isn't matched by somebody else who knows a lot about their area of specialty.  That way, you never do things without either having at least a 2-person concensus or having a detailed analysis of the issue, resulting in better decisions."  Any thoughts?

To the "constraints" comment:  What if the model is "very good at one thing, but can do others" and I distribute those people around the specialties?  It feels like that should even out the constraints, as long as I have the right ratios (enough DBA's, for example).

(Also, I'm still looking for any documentation that anyone has seen.  I feel like I've seen this discussed in a book or a study somewhere, but I can't remember where.)

Nathan Arthur
Friday, March 07, 2003

The main thing with the constraints is to try to balance the flow of work.  You should not try to balance capacity - that will make the situation worse, not better.

As far as books go, Steve McConnell has some advice about staffing.  Capers Jones maintains data on something like 7,000 projects.  His analysis shows that specialists are more productive than generalists, but this factor isn't even in the same league as others, like reuse, at driving up production rates.

You can look at the quality work of W. Edwards Deming or any of the Six Sigma stuff, too.  These pursue quality improvement and process predictability by trying to control and subsequently reduce variation.  If you have one guy producing ten items, they will usually be more uniform than if you have ten guys each producing one item.  My favorite book on this stuff is "Understanding Variation" by Donald Wheeler (ISBN: 0945320531).

pair designer
Friday, March 07, 2003

Gotta disagree about having specialists do architecture, at least in my experience.

A specialist is myopic in nature, the last kind of person you want to make architectural decisions. 

A generalist is usually better at seeing the big picture, and delivering a better architecture.

I've worked with highly specialized people before that have architected things, and when I ask them to explain it to me they immediately jump into the internal details of the code ("We're using j2ee, ejb, xml, jakarta struts, ant, ...").  Ask a generalist and you might get something along the lines of "Well I was talking to Jim in HR and he said if only the old system would do X, Y, and Z it'd be perfect"

YMMV

big bob
Friday, March 07, 2003

Nathan sez: "Never have a specialist who isn't matched..."

I totally agree.  The 2nd person can help the specialist too.  Everyone needs someone with whom they can discuss design decisions, debugging problems, ...  Having the 2nd set of eyes and ears to help in the tough going is essential.  It becomes a productivity multiplier for the specialist.  Sometimes, you get stuck on a problem, and just explaining it to someone else helps construct a framework for analysis.

Anyhow, I think specialization is very often necessary.  It is division of labor and the reason why a team of mecahnical engineers can design a car engine where civil enigneers would fail completely. 

It is essential to have specialists on your team who will contribute to your core competencies.  If your product is a forward looking infrared system, then you'd better have some high powered DSP gurus on your team.  I don't think you'd hire a database guy for the job.  Nor would you hire a device driver guy to write user interfaces.  It takes time and energy to become proficient at something.  As much as possible keep your people honing their skills in what they love.

Nat Ersoz
Friday, March 07, 2003

Just curious. How does the attribute of 'specialization' relate to the attributes of experience level and ability to architect software, and are we assigning 'good' or 'bad' qualities and value judgements to these attributes?


I've never gotten a good fix on this issue. Some companies endlessly debate the fine tuning of these attributes, as though the quantity of competent people excelling in any particular attribute of these three can be found falling off trees and it's the responsibility of the hiring authorities to endlessly nitpick the exact right blend of candidates and to string candidates along almost indefinitely with a smirk that the company is looking for "excellence".


I say - just HIRE someone, and select the candidate(s) that display the most promise (and preferably past track record) of temperamental flexibility.


I believe that a highly competent and productive "can get it" software developer can readily float between being a generalist and being a specialist, and in fact, likes to do this rather than stay at either end of the spectrum for very long.


The 'stupid' developers (or rather, the ones so smart that they are downright foolish) have appeared to me to be the ones that have a lot of trouble moving between the big picture and the detailed low level intensive work that may take weeks or months. I am as infinitely suspicious of the architect that never writes a line of code, only delegates the work, as I am of the career low level dweeb that has a brain aneurysm if you ask him where an end user would see his stuff operating in a finished product.


IE: a guy that only wants to do OS internals for a living and can't even semi fluently describe the value of what he's doing in some kind of real world terms (at least related to the applications that sit on top of it, for starters) isn't a genius - he's more of a childish kind of a boob. When you're being paid the big bucks, you should expect to have to talk just a *bit* once in a while to another real human about your role in things, and defend what you're doing to others. Not constantly, but occasionally and when necessary.


It furthermore seems to me that a highly intelligent person consigned to doing Windows VxDs (or whatever they're called now :-) ) or graphics internals as a job title would go ape s*** out of boredom after a couple of years.

All in my opinionated opinion, which renders the diatribe I just posted highly suspect...

Bored Bystander
Saturday, March 08, 2003

big bob:

When I mentioned a specialist for the architecture, I was only referring to having them doing the architecture within their area of specialty. For higher-level enterprise architecture where disparate systems have to interoperate, the generalist will be needed.

So if you're developing a major J2EE application, it helps to have somebody who knows the ins and outs of how and where to use EJB, servlets, JMS, JNDI, etc.  Without having the specialist with solid experience in Java/J2EE to set direction, the team of generalists will often end up wasting a lot of time doing things in inefficient or error-prone ways, being unaware of the 'gotchas' until they actually 'gotcha'.

But when that J2EE application has to also talk to an IBM mainframe, an Oracle database, and a document management system, you'll also need a generalist to organize how they will all work together.

That myopic individual you referred to is not a specialist -- that person is what they call a "one-trick pony".  There is a difference. A real specialist knows where his/her specialty ends, while still being able to cooperate and communicate with others to enable his/her specialty to interoperate with other technologies.

T. Norman
Saturday, March 08, 2003

Norman, I've worked on some pretty good size J2EE projects, so i'm going to pay devil's advocate and disagree:
Because specialists can sometimes have a limited view of the big picture technology their working with, and because they are so eager to employ the tricks of the trade (for instance EJBs) they often miss the simpler, more elegant solution (of course this is not always the case). 
  In my few years of experience, i've learned that it is better to have a team of quick learners who have a very solid grasp of software engineering and design, and get them to "specialize" along the way.  For instance, I'm not an expert at Entity Beans, but for X project i'll do some extra reading and be the authority on them. 

This is also a better solution when employing XP, because of the shared "code ownership" idea.

Vincent Marquez
Saturday, March 08, 2003

Vincent Marquez said, "...but for X project i'll do some extra reading and be the authority on them. "

I tend to trust authority gained through proven experience a lot more than authority gained only through reading or other kinds of analysis.  These last I view as a supplement to actual experience, not a replacement for it.

Obviously, for bleeding edge work, you have to take what you can get. In general I'd rely on a proven track record more than broad familiarity with the literature.

Experience is what I think a specialist brings to the table more than anything else.  Someone who's seen all the issues, who's learned the hard lessons, who's dealt with all the implementation nasties that spoil all those pretty, abstract designs.  Books and anecdotes just don't let you deliver to the same level.

pair designer
Saturday, March 08, 2003

Remember, we're talking about *good* specialists and *good* generalists here.  Bad ones of either type will mess up your system.  A J2EE "specialist" who insists on creating everything as an EJB when plain servlets or a simple Perl script would be better, is not a good specialist. Myopic behavior should not be an argument against having a *good* specialist.

T. Norman
Saturday, March 08, 2003

BoredB.

I like what you wrote there.  I also tend to think that if someone can do one difficult thing well, they can do anything well.  And for the most part that is true.  But people do need to specialize, I think.  Not that they should be stuck with a specific specialty, but that no one can touch bottom on all aspects of a project.

There is significant overhead involved in becoming fluent in an area - say cryptography for example.  The novice will completely miss things like man in the middle attacks and other vulnerabilities until they obtain more skill in the art.  That learning curve is steep enough that you can expend the time and energy to have everyone beome expert in every area.  At least that's what I think.

Nat Ersoz
Sunday, March 09, 2003

Nat:

Did you mean "CAN'T expend the time and energy ..."?

T. Norman
Sunday, March 09, 2003

Hi, Nat:

>> if someone can do one difficult thing well, they can do anything well.  And for the most part that is true.  But people do need to specialize, I think.  Not that they should be stuck with a specific specialty, but that no one can touch bottom on all aspects of a project.
>> There is significant overhead involved in becoming fluent in an area - say cryptography for example. 

I absolutely agree. But this industry sucks too much to make specialization a prudent career choice. So guess what, specialists are *very* rare.

Mainly, real specialization in something esoteric (good example of cryptographic programming) is bad for your long term financial health, except under very unique circumstances.

That's why I have avoided specialization like the plague. I think most programmers know this intuitively.  So that's why certain "deep" programming talents such as device drivers, kernels and OSs, and cryptography tend to be so hard to find. Companies just want to hire people long enough to do the heavy lifting, then chew them up and excrete them. The main problem is that you're only seen as being valuable in only one capacity, and so you're vulnerable to  being too dependent on one job.

Specialists are generally denigrated by many hiring parties as being "one trick ponies". That's another self justifying HR culture thing.  The person that has specialized within a company will usually be very unemployable on the outside.
As far as those 'unique circumstances' in which specialization is good or worthwhile - this would be the case if you were a consultant with a REALLY good network and you had an industry reputation including speaking and published books and articles. You'd pretty much have to be a consultant because companies are loathe to keep specialists on the payroll, and also because most private business is loathe to allow a mere scum employee to promote himself in the outside world unless it's shilling for the company. Having the high profile is also essential to staying employed as a specialist.


Conversely, there is NO, and I repeat absolutely NO market for an unknown specialist - you can know a subject inside out, but lacking the press or network, you look like a *really* unemployable body to smirking HR people and smirking recruiters. You can say that you know X to the Nth degree til you're blue in the face, even with a track record of actual products, and you will still get the cocked eyebrow that indicates that you're wasting their time. Believe me, I found this out the hard way!


Hence, the conclusion I've arrived at, which is: the very best that private business can hope for is to develop their own specialists on an as needed basis.

Bored Bystander
Sunday, March 09, 2003

BoredBystander makes some excellent points, though they are from the opposite perspective of the subject of this thread.  Specialization, especially deep specialization, is a risky career move.  I don't think it's so much that the industry sucks, though.  It's economics.  In order to build a large market for your special skills, you need a lot of instances of the problem you are so good at solving.  As Joel says, though, software is infinitely clonable.  If someone provides a really good solution to your problem the market will want to buy the solution as a product from them rather than as a service from you, due to economy of scales.  Figure out if you can turn your specialty into a product that you can mass produce and beat them to the punch.

You also have to choose your market carefully.  The problem you solve has to be worth a lot to solve.  No one's going to pay you for your expertise in button design because good buttons just don't have much value add relative to bad buttons.  You have to solve one of those make-or-break problems.

Another problem you run into is the rate of change of market drivers.  Last week's VB6 golden child might quickly wind up this week's maintenance programmer if his specialization is too narrow.  If the time line for developing a master level of something is longer than the period of change of the associated market drivers, you are going to get into trouble.

When you do find a market suitable for specializing in, you have to do like BBystander says:  you have got to develop the market.  Write books, author articles, speak at all the trade shows, and so on.  Also, get good at financial planning.  Instead of a nice uniform series of regular payments, you're probably going to get big lumps of money followed by long, irregular periods of nothing, for the most part.

max
Sunday, March 09, 2003

I agree that specialization is generally not good for the career of a developer, because HR hiring practices are based on the length of the laundry list of acronyms that you put on your resume.  But for an employer, I think specialists are good to have on staff as long as they aren't the majority of your workforce.

T. Norman
Sunday, March 09, 2003

I think specialization that serves *only* the interests of the employer happens regularly in this industry. The 'only' means that, just as stated in the last few replies, highly specialized techies are at a distinct liability in the job market. 


So if your employer grooms you to be a specialist, it's generally to your disadvantage because they can do what they will with you as your list of skills is "shortened". There is a slight chance that the specialization is in something that will be considered desirable in the job market because it's an emerging technology on which you can catch the rising wave if you leave, but  it's usually not the case. As your 'marketable skills' become narrower, the employer has more ability to dictate terms.


A concrete example of this is that just about every embedded systems developer I've known in my area has had (not in the last 2 years, but more like the last 15 years) one *hell* of a time finding stable, non abusive employment. Embedded work is grueling, nitpicky stuff in the best of times, and your 'reward' to do this stuff in my region is to work in a sweatshop, followed by a layoff.

And disadvantageous forced specialization is one big reason I started contracting.  It's a lot harder to stay with the market as a contractor than to veg out in a so called "stable" permanent job, but nobody's had the opportunity to "groom" me into a corner, either.

This whole subject of specialization raises my ire (IE, it PISSES ME OFF!) on many levels because so many techies and programmers naively walk into what is basically a career cul de sac without anyone stating the pitfalls.


And the "we don't like contractors" party line that managers and executives love to state as their mantra is accompanied by an understanding that the only desirable people in their eyes are FTEs that become epoxied to a job description. 


And if business culture wasn't so dysfunctional and abusive by design, there would be an efficient market for dedicated specialists that aren't "world class" by self-marketing standards. Instead, even if you can solve a prospects's problems, chances are that HR and their management will buzzword-nitpick you to death.

Bored Bystander
Sunday, March 09, 2003

Wow, I really like this.  Bored is on a roll - tihis is as good as it gets at JoS from my perspective.  Lappin' it up over here.  Let's change the subject - how 'bout the format of the discussion board...  hell, no.  Don't go there.

Have to say that you hit a nerve over here with the "chewing them up and excreting them out" paragraph.  I gotta think about that a bit.

More later...

Nat Ersoz
Sunday, March 09, 2003

Bored Bystander,

Outstanding commentary!!!!

One Programmer's Opinion
Monday, March 10, 2003

I'm with the rest of you, totally.  BB is on the money.

"Specialization that only benefits the employer" is precisely what's out there.  I mean, occasionally you'll find an employee who wants to learn everything about .NET because it's hot right now, and manages to get his organization to support that...but even then it's usually a result of the organization thinking "well, he might be on to something, and we're willing to risk some of his time to find out."

I've worked for three document imaging companies.  It's not because I'm specializing in the market, it's sheer accident.  Only five years, I can still get out.  ;)

At document imaging co #1, I wrote and trained a neural net AI engine to do OCR, and did some other scientific type stuff in image analysis.  Amazing, fun, cool, deep, theories and hypotheses type stuff.  Loved it.  I'd love to use the principles to do something for myself, I even have an idea that might be worthwhile if I just spent my spare time on it.  But, I'm focused on being a technology generalist.  If I write another AI at work, I'll have "the same year of experience over again" to anyone who doesn't particularly want an AI written, which is (sadly) anyone.  At this point, as much as I encourage document imaging company #3 to pursue ideas that could lead us to writing that kind of stuff, I'd rather support/advise the person writing it than write it myself.

Mikayla
Monday, March 10, 2003

Having had a little time to contemplate my navel on this one...

I whole heartedly agree that hiring managers (often) hire specialists to turn out a job, and then spit out the specialist.  All the while ignoring the opportunity to develop the skills within their already competent technical teams.

But, nevertheless, I'm drawn to specialization like a moth to the flame...  Why?  Its where the interesting work is.  I'm not that crazy about software itself.  The only reason, IMO, that code exists is that something else makes it interesting.  Something else provides the elegance which motivates me to write that ugly multicase exception handler, to reverese engineer that hardware that some moron couldn't manage to write adequate documentation for.  Something a little more noble than realizing that assignment is not initialization in C++.

But when you do get the DMA chain setup, and the packets do scatter/gather, it at least provides the illusion that something higher than syntax got your attention.  I live for that illusion, regardless of the consequence.  Please don't wake me up.

Nat Ersoz
Tuesday, March 11, 2003

Guys, thanks for the kudos. Those vents were founded upon observation of perhaps a dozen (programmer type) people I've known locally who had been jacked around to varying degrees due to their level of specialization. As well as having to undergo some long periods of unemployment myself.


Nat - I hear yah! Part of me feels unclean and common and proletarian to be a generalist and jack-of-all-trades. Believe me, I *want* to dig into image processing algorithms, or the Linux kernel, or scheduling algorithms. But I don't want to dedicate a portion of my career to something that is essentially high risk and provably low-reward.


Back when I specialized, I "reasoned" that I was becoming more valuable in something necessary. I was actually being a "real engineer", understanding first principles, creating the underlying engine that makes everything else work, creating the behind the scenes 'magic' that the mundane heads down applications programmers work with on a black box level, who are fundamentally illiterate about the real guts of the product.


It takes a very sharp, quick, bright person to be an excellent specialist. It also requires attributes that are signatures of good character, such as a lengthy attention span, attention to detail, depth of understanding, and hard work. A big part of me considers low level specialized work the "PhD theses" of our industry; the application development in RAD languages is the grade school stuff by comparison.


But this kind of idealism kind of vaporized years ago for me when I was a Windows C++ 'specialist', used to writing very difficult code to solve esoteric problems, and chasing endlessly after really low  quality work just to work really, really hard to get things done. I would hear about Oracle DBAs essentially writing scripts for applications, billing at 2x my rate. I had also bumped into application developers who had the ear of management and who were known as "go getters" creating real value. The point is, it started to seem as though the lighter weight the work and the developer, the more marketable it/he/she was.


The problem with being a specialist is that executives and managers usually don't have the brains to understand that their products or services may rest upon a foundation of highly specialized guts.  If the logical connection between delivered work and revenue is more than once or (at most) twice removed, then expect an executive and the system at large to belittle the work. Even if everything the company is doing rests on the work and depends upon it.


So in a sense, being a software or design or engineering specialist has a lot in common with being a garbageman: it's a low regarded role that may be pivotal in making things work smoothly.

I think these recent IBM tv spots that show a table of suits talking about magic pixie dust for their servers or reality detection devices - are dead spot on in illustrating the shallowness of the kind of thinking I'm lamenting. The suits want to flip a switch and see money flow in. They find it painful and repulsive to consider that someone with real brains, much smarter than they are, had to work their ass off to create the technology they are attempting to sell.

Bored Bystander
Tuesday, March 11, 2003

*  Recent Topics

*  Fog Creek Home