Fog Creek Software
Discussion Board




Problem Solving Skills

I'm about to recruit some programmers for my small co.  For all good reasons, I only want to recruit people with very good problem solving skills.  I plan to give them both written and oral tests to determine how good they are.  Anybody who has done this before and would like to share his experience?  What  kind of questions did you ask? 

I must admit I have not been very keen on this in the past but I think I can now afford better programmers.

Ling
Monday, November 24, 2003

You can now afford better programmers?

So you will lay off the ones you have and hire new better ones?  IMO strong programmers are very picky when it comes to management and how thigs are run.

So its not really about asking the "right" questions, but to provide a good work environment.

Patrik
Monday, November 24, 2003

PS.
Here are an earlier thread discussing programming related questions:

http://discuss.fogcreek.com/joelonsoftware/default.asp?cmd=show&ixPost=60273&ixReplies=72

Patrik
Monday, November 24, 2003

So you should wait until you can hire the best people that you think you'll ever want and never hire cheaper, not as experienced people?

I can't afford to hire Joel's amazing superstars! I guess I'll just not hire anyone at all.

Fred
Monday, November 24, 2003

If you want to see how they really work, give them a short assignment and allow them to complete it in their free time (a week/weekend or so).

Review the project they submit and if you like their solution, invite them back for a second interview.  During the interview, have them explain what they did and why they did it. 

Don't make the assignment too lengthy, but make it complete enough so that it parallels what they would be doing in their day to day job.

Why not try the milk before you buy the cow?

Guy Incognito
Monday, November 24, 2003

It's not about buying "amazing superstars", it's about buying the best programmer for the salary that you can afford to pay.  There's nothing wrong with being a bit choosy. 

Guy Incognito
Monday, November 24, 2003

Guy Incognito:  You must know Moe Fuzz.  (That'd be a reference to the movie "Tapeheads," for the other 99.999% of the western world who didn't see it...)

Personally, I didn't mind doing a few quickie tests as part of the hiring process, but if someone asked me to solve a problem related to their work on my own time at home *before I even got a second interview*, I'd see that as a sign that said employer had really bad boundary issues.  There've been prior posts here on this topic, and IIRC this particular practice was not terribly popular.  (=

Besides, you can learn all the same things by asking them to explain some pre-existing source code -- plus, if you're not familiar with the problem domain, you get to evaluate their communication skills.

Sam Livingston-Gray
Monday, November 24, 2003

We'll it's a buyers market right now, so why not try before you buy?

Do you realize the cost of hiring and firing someone?

Gal Incognito
Monday, November 24, 2003

Where is this "pre-existing" source code coming from?  Did the interviewee write it?  I do like the idea of an interviewee bringing in a "programmer portfolio", but I haven’t seen too many of these.

Gal Incognito
Monday, November 24, 2003

The first place I interviewed with had me write code for approx 4 hrs, to solve a real problem in their code. This fix later made the production release.

I didnt at the time consider this bad practize by the employer, since I was a bit young at the time and happy to be writing code that had some "real" use.

At the second place, we walked through the application and as a result from my questions we looked at some code in the application as well.

These were both really small shops (5-10 ppl.).

We later introduced fictious assignments for the people that we interviewed with, so we had them write code for approx. 1 hour or so.  That gave good grounds for judging the guy.

Patrik
Monday, November 24, 2003

"Do you realize the cost of hiring and firing someone? "

That is one of the reasons why there is so many staffing firms out there and "contract-to-perm" employment has increased in recent years.

As for test taking, I am against it but I can understand why many employers do it and why many programmers seem to recommend it.  Imo, test taking is good for two types of situations:

* Heads down coding positions

* Situations where the person you are hiring needs to have extensive and specific technical experience (i.e. you hiring someone specifically because they claim to have X number of years using Y technologies and the Z programming language).

IMO, many business application developers who take offense to coding tests do so because they were probably doing a lot more than just writing source code at their last job or assignment.  For example, their previous employer might have demanded that they work on their soft skills, such as communication, getting to know the decision makers, learning more about the employer's business domain -- so this is where they spent a considerable of their recent time doing.

One Programmer's Opinion
Monday, November 24, 2003

http://www.techinterview.org/ ?

www.MarkTAW.com
Monday, November 24, 2003

Ling,

This seems like the kind of thing that somebody with good problem solving skills would be able to solve for you.

:)

Joe Blandy
Monday, November 24, 2003

Or a problem that someone with good delegating skills would post on the internet somewhere....

Danil
Monday, November 24, 2003

Request that programmers bring in a portfolio of code.  Perhaps edited samples of their two latest projects.  Sometimes they won't be able to bring in the code (their employer won't let them), but some may have code they can showcase (be it contract work where they own the code, or OSS, etc).  Be sure to ask critical questions about the code they present.  Why were certain choices made, etc.

If you do have sample problems, make them 1-2 hours in length, advertise this fact so they know they will be expected to code, provide them with internet access and several books on the subject (no one codes in a bubble), and offer them a quiet place to code.  Make the code as simliar to a production issue as you can.  After all, you should test them on what they will be doing.

And have another employee (a trusted programmer) take them out to lunch.  Get a second opinion on the personability of the person.  That really counts in small shops; it counts everywhere for that matter.

Lou
Monday, November 24, 2003

"If you want to see how they really work, give them a short assignment and allow them to complete it in their free time (a week/weekend or so)."

So *that's* why we see all these 'help me with my code' posts on the weekends here on JoS!! :)

Dennis Atkins
Monday, November 24, 2003

"We'll [sic] it's a buyers market right now, so why not try before you buy?"

Isn't that called "situational ethics?"

Sam Livingston-Gray
Monday, November 24, 2003

One thing to add to all this: NO WRITTEN TESTS FOR TECHNICAL ASSESSMENT unless you buy someone else's test. Do NOT formulate a "skill assessment" test of your own.

If you do this, you will look like a two bit dork. In fact, if one does so they probably ARE a dork.

I know a company that does this. They do this because the owner is a control freak. I know two senior people who have interviewed there and they both regarded the practice as exceptionally mickey mouse and demeaning.

Bored Bystander
Monday, November 24, 2003

I applied for a job once and they gave me an IQ test out of a magazine. It was pretty simple and I'm a smart guy, and I was surprised when they said I got one wrong. Whatever you do, don't do that.

Bored Bystander is right, this problem has probably already been solved (pun not intended). Find out the standard solution, see if there's a standard test, and implement it.

www.MarkTAW.com
Monday, November 24, 2003

Geeeez.  If I'm going to hire a writer I'm not going to test him for spelling or grammar etc. If I'm going to hire a proofreader I might do that. When I hire a writer I'm going to want to read what he has written. A programmer must have written some programs so I will want to see those. 

I have had places ask me questions. Amazon did that. They asked me questions that anyone in their first year of CS would know, nothing that I ever encountered in 5 years of production programming for a web environment. I just told them I didn't know. I didn't get the job. Oh well, I was not too impressed with the stupid questions. They never asked to see any code.

Reading someone's code will tell you a lot ... if you are smart of enough to read between the lines. If not, just ask some parlor questions.

Me
Monday, November 24, 2003

Me (+1 Insightful)

Guy Incognito
Monday, November 24, 2003

"One thing to add to all this: NO WRITTEN TESTS FOR TECHNICAL ASSESSMENT unless you buy someone else's test. Do NOT formulate a 'skill assessment' test of your own.  If you do this, you will look like a two bit dork. In fact, if one does so they probably ARE a dork."

Care to explain *why* you feel this way?  I've spent several years refining my tests (and had some excellent help from people here), and they have proven to be excellent predictors of success and weeders of fakers.

Grumpy Old-Timer
Monday, November 24, 2003

In many cases a programmer will have absolutely no legal ability to show you the most important programs he or she has worked on, since they are owned by former employers.

Mr. Fancypants
Monday, November 24, 2003

>> Care to explain *why* you feel this way?

The main problem I have with developing customized written tests to gauge problem solving aptitude on the part of the candidate is that (IMO) it takes a knowledgeable specialist to develop a good test.  Just because you're a developer and know programming does not mean that you have any ability whatsoever to develop a meaningful test in that subject. That certainly goes for me, too. 

Given the choice between written tests, the candidate's own work, or on the spot "real" programming tests, I would choose either or both of the latter two.

Unless you have experience developing such tests successfully in a professional capacity, you'll probably screw it up, create a test that tests the wrong attributes, and/or create a test that alienates your candidates. I am claiming that you will in all likelihood obtain meaningless data from the test.

The test that one of my clients uses in hiring developers is heavy on generalized verbal logic tests, because that's what the owner values and knows. This test is uniformly regarded by his people as nonsensical and was only tolerated by each of them in order to get the job. Also, the company in question has had a track record in hiring programmers which has been little better than random chance - some hirees have been absolutely useless. However, each one "tested" out OK. (except me - I had a portfolio of past work and the issue wasn't even raised.)

I guess a written test would be OK if you had the means to test the correlation between candidate scores and candidate performance. That would require a LOT of data points. Well beyond the resources of a small company.

I will make just one exception. A very clever "paper" test that generated a lot of bile when presented on a BBS some months ago was to vet candidates by making them list all syntax and obvious declaration errors in a one page source code listing. This test (decried as "demeaning") wound up smoking out people who claimed development experience who could not identify even simple basic syntax problems...

Bored Bystander
Monday, November 24, 2003

Employer created tests are fine as long as they are made by an actual developer and as long as there is some reality check to make sure the system works. Microsoft uses them. Do they have a problem hiring good people? Joel uses test questions he makes himself. Is this a problem?

The tests are to rule out pretenders and to flag a possible exceptional person. Only a fool would use the scores to directly rank candidates with similar scores.

Dennis Atkins
Monday, November 24, 2003

I guess it depends on what you're trying to test for. Programming skills tests are probably easily employer created. Basic logic & problem solving as independant skills, on the other hand, are a different story.

Joel is probably the exception to the rule, he's also extremely well read, exceptionally intelligent, a good writer, and has an interest in things outside of purely doing his job, like psychology and so forth.

Have I mentioned techinterview.org?

www.MarkTAW.com
Monday, November 24, 2003

Whoa, Mark! Are you going for an interview with FogCreek later in the week or what?


Tuesday, November 25, 2003

"...it takes a knowledgeable specialist to develop a good test.  Just because you're a developer and know programming does not mean that you have any ability whatsoever to develop a meaningful test in that subject..."

We can agree to disagree on this point.

I have developed a test that lets me do one thing with extreme accuracy: differentiate people who know some C++ from complete fakers who read a book yesterday and think they can bluff their way through an interview with a PHB.  In my experience, this eliminates 95% of the candidates, allowing me to focus on the rest.  (I know that the 95% figure is hard to believe, but unless you've interviewed C++ candidates, you'll just have to take my word for it.  You wouldn't believe the people you get...)

Would my test allow me to say, "Bored" knows more C++ than "Grumpy"?  Of course not - that's silly.  It will let me say, "Bored" and "Grumpy" both know enough C++ to be worth further interviews.  And tht's all I need.

Grumpy Old-Timer
Tuesday, November 25, 2003

I think a simple, first-order skills written test would have its place. IE, something to determine whether candidates have ever developed in the language du jour. (like the test I mentioned above where candidates are asked to spot obvious syntax and declaration errors in a page of source code.)

I just think that any goal more ambitious than flushing out the total losers is unrealistic because the skills needs to construct such a test are too specialized and subtle.

IE, it's unrealistic to develop your own test to separate "junior" from "senior" developers.

Bored Bystander
Tuesday, November 25, 2003

Nah, I'm not. I just wanted to say that your typical paper pusher isn't qualified to write interview questions because he'd never given it any thought or read up on the subject.

Meanwhile it's probably been on Joel's mind for a long time, and he probably reads considerably more in related subject matter than a random person in Corporation X.

Especially if what you're testing is outside of their own job description - i.e. Problem Solving (which is something they're not likely to have given much thought to) relative to a strict programming test.

I know that I wouldn't want to have to write such a test, though I'd be willing to attempt it if I was given enough time to mull it over and read up on what problem solving actually entails. This is really one of those "In the shower AHA!" situations.

www.MarkTAW.com
Tuesday, November 25, 2003

*  Recent Topics

*  Fog Creek Home