Fog Creek Software
Discussion Board

"Less than expert" Technical Professionals?

I'm having growing unhappiness with the lack of technical expertise I see in the IT industry. As I advance in proficiency, I keep thinking that there is so much more I have to learn - I feel like a journeyman, thinking there are true masters out there.

But increasingly I'm thinking I *am* a master in a sea of journeymen...

The most recent thing that's made me think this - I deal with an EDI document company. When we signed the contract with them, they guaranteed they could handle XML. Well, it turns out "handle" means "barely cope with" - they only "handle" it by writing it out as a flat file, which means through integration we kept having to harp on them about violating XML standards (and they still can't handle empty tag shorthand: <likeThis /> )

Okay, so their salesmen were a mite overeager, fair enough. But even in EDI, where they are supposed to be experts, we keep having to teach them stuff. The latest is dealing with 997 acknowledgement documents, where it seems they have no concept of how they are supposed to work. Last week their senior mapper commented how cool some of the stuff we're doing is, when all we're doing is following the specification.

And so it goes. DBA's that don't understand stored procedures; sysadmins who don't understand DNS caching; programmers who can't grasp recursion...

Is it just me? Are my expectations too high?


Sunday, March 2, 2003

I have definitely noticed this.  I work with a bunch of Perl programmers who don't understand variable interpolation in double quoted strings.  Not to mention all variables are global in all of our scripts because no one understands scoping.  These practices are actually encouraged by the people in charge.  They don't want anyone who never made it past page 10 of Learning Perl to worry about learning anything.

I think the problem is too many people become technical professionals because of the money.  I have yet to work with someone who does it for the money that is any good at it.  I guess it doesn't hurt that most people have low expectations for software.

Sunday, March 2, 2003

>>> I think the problem is too many people become technical professionals because of the money. <<<

They wouldn't be working at a technical job if someone hadn't decided to hire them.

This thread sounds a lot like others we have had.  The people who post to this site are technical professionals who get frustrated by seeing such things.  But the people who make the decisions that create such a work environment are management people.  It would be very interesting to get a few of them to tell us why they make the decisions they do, but I don't think we're going to hear from them.

Sunday, March 2, 2003

You get what you pay for. Companies hire the anyone that will do the job for the least amount. These Technical Professionals are really just task doers. If you have competence and want to make money go contracting. If you have competence and want to do real interesting work try getting a research programmer position at a national lab or university.

Sunday, March 2, 2003

What is a research programmer?

The Real PC
Sunday, March 2, 2003

>What is a research programmer?

A programmer that works with a team of researchers who works on some unsolved cool problem. Think Palo Alto Research Labs or the likes of it...

Thats my take :)

Sunday, March 2, 2003

You get what you pay for. Companies hire the anyone that will do the job for the least amount

This is -NOT- always true.

Personally. I think a lot of it has to do with core competancies and insecurity.

Formula for failure:
If a company has insecure management ("B People"), they will hire people who are not as good as they are ("C People") in order to feel more secure.  At this point, the company is just screwed. :-)  Joel Spolsky wrote an essay on this, so it must be true. :-)

Forumula for success:
By the same token, if a company is founded by coders or the companies "strategic edge" involves technology - and the management of those areas gets the best and the brightest - you might NOT have a problem, because they view Technology and as INVESTMENT, not a cost center.

This is especially hard to do if you are coding internal apps.

It's easier to do if you are selling the stuff, but mistakes are still common.  My conclusions?  In order to be recognized as exceptional, and get paid for it, you need a few conditions:

(1) Be able to articulate the value of what you do, and why it's more than joe incompetant,
(2) Have a management team that intutitively "gets" technology and views it as an investment,
(3) Have a work environment this has pride and recognition, instead of fear and insecurity.

In any event, good luck.  In my experience, birds of a feather flock together.  Perhaps you've just been flying with the wrong flock ...

Matt H.
Sunday, March 2, 2003

This is especially hard to do if you are coding internal apps.

--- When i wrote this above, I ment it in the context of "articulate the value of what you are doing"

In other words, if what you do this year will sell for $1,000,000, it's pretty easy to understand the value of hiring the best and brightest.  If you're doing internal apps, it's harder to "prove" the "worth" of hiring the best and paying them what they are worth.


Matt H.
Sunday, March 2, 2003

I think there are several factors in confluence:

First, software has become a mass market commodity.  So the hiring needs of businesses far exceed the gene pool's and the educational systems' ability to produce competent candidates for software development. It's one thing if a need for 10x the number of plumbers we currently have all of a suddenly became an issue for economic growth in the next decade. Instead, our economy in the 90's needed masses of people to do work that only a small and gifted percentile of the population is capable of doing competently and 'getting'. The ability to think and reason abstractly doesn't grow on trees.

Secondly, like any hot growth industry, the computer industry has become a hotbed of inflated egos and a prevailing attitude that anyone who is any good acts like they're God. Dripping arrogance has become pretty much an expected mentality in this field. So you often can't distinguish the really good people at first glance from the ones that are only talking themselves up who can't perform.

Third - the commodity nature of programming and the ego factor have created a living example of Gresham's Law - good currency chases out bad. By that I mean that it doesn't really matter how good you are, it's more a question whether you are well connected enough to be heard above 10 or 100 other people with the same nominal 'skills'. It doesn't matter if you can design and engineer effectively at a high level, and if you can adapt quickly, at least when it comes to convincing someone to hire you. No, hirability anymore comes down to a number of petty factors such as: not intimidating the hiring managers; pure personality match; having the right alphabet soup skills, the right language versions, etc; and being in the right age group so that you won't cost the employer too much in health insurance, time off for necessary family matters, etc.

Except for tiny pockets of virtue, this industry excels at crapping on experienced people that know how to achieve results. It doesn't seem to make any sense on the surface, but look at how many programmers of age 45+ are around.  If experience were valued, you wouldn't have the widespread habit of referring to anyone with years of experience as an old fogey loser who is tainted by experience with outdated technology. The demographics pretty much point to a meatgrinder that treats individuals like trash and wants only a PC-acceptable template person.

Bored Bystander
Sunday, March 2, 2003

>> Gresham's Law - good currency chases out bad

ACK, *I* should be chased out.


Gresham's Law - bad currency chases out good.


Bored Bystander
Sunday, March 2, 2003

If "A"s hire "A"s and "B"s hire "C"s, who hired the "B"s in the first place? :)

An issue here is mentorship. I see people coming straight out of colleges and other training courses, etc being hired as programmers. Fine, no issue here, the issue is that some of these people are "let loose" without any/enough guidance from somebody with enough experience (AND the skills to guide them) to teach them best practices, potholes to avoid, etc. These unchecked bad habits become standard practice, as these developers get "more experience" (ie. they clock up more years), they get more respect/power/influence over projects - in situations like this "years accrued" is not necessarily a clear measure of ability.

Walter Rumsby
Sunday, March 2, 2003

Your company should have broken the contract with this EDI document company. Your company was mislead and the business relationship should been amended or ended.

Philo wrote, "Is it just me? Are my expectations too high?"

Matt H. comment about differences between people who do internal application development and maintenance work vs. people who do commercial software development work was spot on.

Technical experts/specialists are rarely hired (on a FT basis) by non I.T. firms such as insurance and manufacturing companies.  Also, large consulting firms tend to hire more business and political savvy developers. Unfortuantely, many of these programmers can't code their way out of paper bag.  Why?  These employers want people (or used to anyway) who are:

* very adaptable
* capable of dealing with office politics
* capable of communicating with various end users
* have the potential to assume a managerial position.

Philo wrote, "But increasingly I'm thinking I *am* a master in a sea of journeymen..."

What you think is un-important. What counts is what your employer/client thinks of you.

Masters typically are self employed.  Why?  The reasons vary.  Some employers can't afford them, some employers are unwilling to pay them want they are worth, .....

One Programmer's Opinion
Monday, March 3, 2003

FWIW the "surrounded by incompetence" feeling is also reported by most of the people I know outside of IT.
Could just be most people are able to recognize incompetence in others but not in themselves.

Just me (Sir to you)
Monday, March 3, 2003

It's a vicious circle.

Hiring people - not just techies - is hard; you have to evaluate a whole bunch of different competencies, decide whether the person fits in the organization, look for telltale signs of genius, insanity and anti-social tendencies, and all - usually - in the space of just a couple of meetings.

Usually, there's a distinct time pressure - you have to get someone to replace a key team member, or you need to have enough staff to complete a key project, or you are living a political hell which forces you to use your headcount allocation before a certain date. You may be working in an organisation which regards humans as "plug-compatible" components - a developer is measured by years of experience in a particular set of technologies, rather than other, harder to measure criteria.

If those pressures force your hand in the early stages of putting a team together,  you can end up with a team without "pushers" - the people who explore the boundaries of the product, its design, the way its developed etc. In my experience, the only way individuals grow as professionals is when they are exposed to new ways of thinking or doing their job.

Without those irritants in the team - people who question the status quo - the team is likely to remain on a plateau of ability. And guess what - it's very hard to hire people who are "better" than the average in a team, because nobody wants to hang around a bunch of dolts. So, you tend to end up with stagnant teams hiring people much like themselves, and never really making the leap to the next level.

Monday, March 3, 2003

"Could just be most people are able to recognize incompetence in others but not in themselves."

There was a major study released within the last few years that shows exactly this.

I am well aware of my shortcomings, and constantly feel like my abilities are not what they could be (this is why I push for and welcome peer reviews - I want to learn from my mistakes).

And when confronted with things like the 997 issue, I'm the first to say "Hey, am I doing this wrong? Please tell me if I am." Only problem is that it seems more and more like I have to explain the problem first before I can even get an answer.

The personal misgivings about my own abilities are why I posted this note in the first place. ;-)


Philip Janus
Monday, March 3, 2003

Philip, it's just that there's a bunch of Boneheads in every field.  To quote Sturgeon's Law "80% of everything is crap"

Greg Kellerman
Monday, March 3, 2003

Actually, Sturgeon's Law states that "90% of everything is crap."

Brent P. Newhall
Monday, March 3, 2003

Wouldn't go expecting anything out of new CS grads.  Last year I was in the labs looking at a girl's source code...

She was cutting and pasting pieces of Java code (We ONLY write in Java.), then connecting them with switch statements.  All this to write a four value modular add.

  switch x {
              case 1
              case 2
... etc.

The courseware usually goes through four or five bug-fixes after its released.  Our lectures don't even appear to have learned about the collections framework yet ...

Sigh, at least I'm being prepared for the real world :-)

Thomas Barker
Monday, March 3, 2003

I know that many of you feel mentorship is one of the best ways to learn how to be a great programmer, but how do you suggest a CS grad or student get involved in situations where they can learn from people better then them? Won't most of them not want to waste their time with someone who is still learning?

Phil Larson
Monday, March 3, 2003

Won't most of them not want to waste their time with someone who is still learning?

I don't know about everyone else but I don't feel this way. My friend's little brother is a CS student. He's taking his 2nd programming course which is in Java. The professor expects the class to write some code but no one in the class can even get their environment set up. (The previous professor taught them theory only). Granted Sun's tools can confuse people, especially when classpaths & packages are involved.

Anyway, I helped him get his java env set up and taught him some basic concepts that his professors left out (Like what an interface really is and what the advantage is to using them). I enjoyed it cause I could talk to someone outside of work about programming (most of my friends are not programmers), and when he finishes and gets going in his career I will feel proud as to have helped him out. (even if it's just a little).

Monday, March 3, 2003

IMHO, there are three great ways for a student in college/university to find a mentor:

1) Make friends with professors.  There are bound to be one or two that are worth your time.  Read their webpages, look into their work, and e-mail them about it.  Ask questions.  See if you can get involved in their research.  Tell them that you're looking for people to learn from.

2) At my university, a number of my classmates were working professionals who had returned to school.  You can strike up conversations with them, see where they work and what they do, and ask them about it.  If you're stuck on a problem (class-related or on a hobby project), approach these people for advice.

3) Look into a part-time job in your field.  "Part-time" can mean a couple of hours per week.  Tell the recruiter at your university that you're looking for experience in your field, but don't have lots of time to pour into another job.  There's a chance you'll find something.

Brent P. Newhall
Monday, March 3, 2003

While I understand that not everyone can learn things the same way, I don't understand why many people refuse to put forth any effort to learn things on their own.  There are thousands of books and magazines on just about anything you could want to know about.  Read up on things and try them out for yourself.  Everyone makes mistakes and I wouldn't expect a fresh graduate to know all the ins and outs of software development (or anyone for that matter), but there is definitely a lot that can be learned on your own.

I never went to college and learned basically everything on my own through reading and experimentation.  I have also never been sent to a training course by one of my employers.  Maybe this is just the way I learn well, but it seems as though the better programmers I have worked with learned this way as well.  I am not saying you should not go to school, but some things must be learned on your own.

As far as looking for a mentor, I would look into any user groups in your area.  Unfortunately I think these types of things are less common for Microsoft products and technologies.  There are plenty of them for things such as Perl, Linux, and Java.  These groups often give presentations once a month and you would have a good group of people to ask questions of.

Monday, March 3, 2003

I can sort of relate to that.  In my experience at uni, very little learning comes out of lectures.  It's generally a series of "Ahhh! That's how you do it." moments.

I guess on CS courses we get pumped with theory, then hit a brick-wall later on because we can't do practical things [GUI, event-handling, pointer arithmatic].  Self-taught people learn all the practical stuff, then hit brick-walls trying to do theory stuff [big-o notation, spanning-trees].

Good people fight past the brick-wall, others just muddle along.  Since the people who hire us (generally) don't know what we're doing (otherwise they wouldn't need us), muddling along is far too easy.

Thomas Barker
Tuesday, March 4, 2003

Since someone refered to it, here is a link to the article:

Unskilled and Unaware of It: How Difficulties in Recognizing One's Own Incompetence Lead to Inflated Self-Assessments

People tend to hold overly favorable views of their abilities in many social and intellectual domains. The authors suggest that this overestimation occurs, in part, because people who are unskilled in these domains suffer a dual burden: Not only do these people reach erroneous conclusions and make unfortunate choices, but their incompetence robs them of the metacognitive ability to realize it. Across 4 studies, the authors found that participants scoring in the bottom quartile on tests of humor, grammar, and logic grossly overestimated their test performance and ability. Although their test scores put them in the 12th percentile, they estimated themselves to be in the 62nd. Several analyses linked this miscalibration to deficits in metacognitive skill, or the capacity to distinguish accuracy from error. Paradoxically, improving the skills of participants, and thus increasing their metacognitive competence, helped them recognize the limitations of their abilities.

Just me (Sir to you)
Tuesday, March 4, 2003

Any books that are used in CS courses are available to the general public usually.  I wouldn't be so quick to assume that self-taught people don't understand the theory as well.  Some of us do this stuff because we really enjoy it.

Tuesday, March 4, 2003

While I will fervently agree with you that there are a lot of self-titled "experts" that are, well, far from experts, there is one aspect to this that paints it a different colour: The software engineering field is absolutely massive, with probably more required information, with the greatest rate of change, of probably any field out there. NO ONE, and I mean no one, has total knowledge of the field, much less one small area of the field, much less one small area of one small area of a field. :-) Couple this with the fact that few firms are willing to keep someone on staff to just be the "XPath Guy", and you have a lot of general practitioners who are called upon to offer advice and wisdom on such a breadth of topics they can't possibly remain experts at all of them.  On the flip side you'll get a short-term expert: Someone who'll read an article on a technology and will then go around quizzing everyone, people who likely haven't touched the topic in question for months, on specifically the topic that they just consumed. I personally work with over a dozen programming languages regularly on a variety of operating systems, using a variety of RDBMS': I think I can be excused if I can't recall off the top of my head how the syntax of a table resultset user defined function in SQL Server 2000 is created.

The primary skill of a software developer is not rote memorization of a niche technology, but rather dedication and problem solving skills.

Jimmy Chonga
Wednesday, March 5, 2003

*  Recent Topics

*  Fog Creek Home