Fog Creek Software
Discussion Board




Is Microsoft's Interview System Crap?

If I remember correctly, Joel has been pretty keen on the IQ quiz style interview but according to this review

http://books.slashdot.org/article.pl?sid=03/04/22/224224

The book he just mentioned "How Would You Move Mount Fuji? Microsoft's Cult of the Puzzle" suggests that that system is crap and doesn't really find good people at all.

So, Joel, in your copious spare time, read the book and tell us if you think he has a point :-)

Gregg Tavares
Saturday, May 03, 2003

The thing I dislike the most about interviews is having someone glaring at me waiting for an answer that requires a bit of thought. And then as soon as giving up on the question being shown the door. Harsh.

Just beforehand I had passed an hour long written programming test and in my first interview pointed out that the interviewer's solution to his own problem was inefficient and I knew a much better way, which I told him about.

On my way home without any pressure I worked out the answer.

Bagpuss
Saturday, May 03, 2003


Yes and no.

I've interviewed three times on campus, resulting in two rejections and one job offer. The difference between the two bad experiences and the one good one? In my opinion, it was luck.

The first two times, I interviewed with people who were severly lacking in the personality department. I was going for the software design engineer position, and the questions I got were half brain teasers, half classic (although not extremely common) algorithms to implement in C. We always went straight to the problem(s), rarely talking about my experience or my thoughts on X. I solved all my problems, although sometimes it took me a while (I really suck at probability problems); the geeks I was interviewing with got kind of impatient if I didn't make headway very quickly, even though I made sure to discuss my thoughts, etc.

The third time was a dream. I actually got to interview with teams I was interested in (instead of some random assignments), and while I got to do plenty of coding, I also got to talk about topics in programming languages and design with some pretty cool people. I felt like I got a lot of respect. Maybe it was because I was talking to people who were interested in the same areas I was (compiler/prog. lang. design)... but it seems to me that the guys in my previous interviews should have at least been able to have a nice conversation about coding. There's only so much you can communicate by implementing a n-ary tree (I even made it generic - no respect!)

Anyway, the whole point is that it has a lot more to do with who you meet than Microsoft would like you to believe. They have the reputation of using puzzles to test candidates, but all the anecdotal evidence I have heard suggests that a mix of good ol' charm, solid background knowledge, and a little luck is what gets people in the door.

Which isn't to say that's a bad thing... I think a lot of their opinions are good (they ignore GPA, and they like to hear about personal projects rather than classes), but some of them suck (don't get me started on the term "super smart" - ever read Show-Stopper? "Let's hire super-smart kids who barely know how to code and let them write an operating system! Anyone who's super smart can learn to code AND learn our huge APIs AND write a kernel! That's why they're super smart!" Right...). That company is just weird.

Dan J
Saturday, May 03, 2003

It depends on what you're testing for.

Written exams and oral exams test for different things. A large part of many jobs is face-to-face contact, and it's good to know whether or not you'll perform during meetings.

www.marktaw.com
Sunday, May 04, 2003

It's not total crap, but it's not very good. At this point, like most long running shows, it's become a parody of itself. Having people write code and solve puzzles is OK, but there are other things that would be better.

The problem is that the people at Microsoft that do the interviews are not skilled interviewers.

Clutch Cargo
Sunday, May 04, 2003

I would agree with Clutch, that after a time, once everyone knows the game its effectiveness wears off.

As an example - I interviewed with one company who, during the interview, had two gentlemen sit down at my left and right, facing each other.  Both pulled out newspapers and started reading stock quotes to each other.  All this time the interviewer is asking rather standard questions, and then would throw in a "What was AT&T's close?" or the like.

What they're testing is one's ability to focus amid seeming chaos while being able to pay mild attention to the environment.  I thought it was quite a good interviewing process, but it caught me quite off guard and I didn't fair as well as I would have liked (but I hear that I did about average).

The next day I interviewed with a different company who did the same thing.  Of course I did much better.  Was it an effective measurement of my skills?  Likely - but was it an effective leveling tool across all candidates - likely not.  Those who had seen the game before were better prepared than those who were new to it, and the interviewers had little way of determining prior experience.

When you're the only one doing your quirky interviewing process its unique nature gives you insight into your applicants.  As it becomes well known or used elsewhere it turns into a relatively meaningless experience.  No interviewing process lasts forever.

Lou
Monday, May 05, 2003

Lou, both companies are idiots.

Your experience shows that that kind of attention can be trained.  An interview that only finds what someone can already do fails to discover the most important thing which is what they may grow to do.

Simon Lucy
Monday, May 05, 2003

Lou - maybe you would have fared better if you had asked them to STFU since you were having an interview.


Monday, May 05, 2003

I'm a hiring manager at Microsoft. I have interviewed around 20 candidates, and have never asked anyone a "puzzle" question - in fact, I've explicitly been told not to, both informally and in formal interviewer training. I'm certain such questions are used by some interviewers - these were the prevailing style some years ago, and are gradually falling out of favor.

I ask people fairly straightforward design and coding questions, and have them write out their thinking on the blackboard. What I'm trying to find out is whether they can go from a problem description to an algorithm, and then to respectable code.

I have my doubts about this interviewing system, chiefly about "false negatives" - people who don't code well under time pressure may be unfairly penalized. However, I think the real system as I've seen it applied is a lot better than the caricature of the MS interviewing system discussed in the Slashdot article. I'd agree that your experience will differ greatly depending on who you interview with - not all of our programming staff are clones, believe it or not...

Pseudo Nym
Monday, May 05, 2003

I think that a bit of confusion about interview puzzle questions good/bad is that there are different types. I'll classify them in three ways:

- "A-ha!" questions. These are the real puzzle ones where there's only one right answer and you're either going to "get" the puzzle or not (or have read it on some web site). Whether you get the right answer is more important in these than your thought process. These are the ones that everyone agrees are stupid and don't really test anything.

- "figure it out" questions. These are the ones where you have to figure something out, but there are multiple answers and the interviewer is more interested in your thought process. For example, "how do you move mount fuji" and "how many gas stations are there in LA". These are better, but still questionable because the interviewer often has a bias towards certain types of approaches.

- "design" questions. These are the ones where you are asked to design something. Again, the interviewer is looking for how your though process. Specifically how you approach product design. For example, the spice rack for the blind question.

Bill Tomlinson
Monday, May 05, 2003

Also note that some interviewers ask these questions *not* to get the "right" answer, but because they want to see your approach and get a feel for a personality.

One of my previous employers asked me a few programming questions, at least one of which I simply couldn't solve.  I was hired, and I think it was because I showed that I was willing to admit I was wrong and that I tried to puzzle through a problem for awhile before giving up and saying I'd have to look it up.

So, the questions can have utility as an indicator of personality.  Peering at somebody as they try to answer a question, then ushering them out if they get it wrong, is simply a bad practice.  That has nothing to do with the viability of asking questions.

Brent P. Newhall
Monday, May 05, 2003

When I am interviewing a candidate I care deeply about three things:

1) Are they passionate about technology?
2) Are they smart?
3) Can they write code?

Brain teasers tell me about none of those things. They only tell me whether the candidate is good at brain teasers.  "Good at brain teasers" and "smart" may well be correlated, but there are too many smart people who are lousy at brain teasers.

Microsoft's interviewer training now recommends against using brain teasers.  Instead, they recommend coming up with good _technical_ questions which are _relevant_ to the job.  I therefore always ask technical questions which are (loosely) based on problems I actually had to solve when I was a fresh out of school coder -- those are the sorts of problems we'll be throwing at new hires.

I know of no way to test (3) without actually getting people to write code in the interview.  That may put people who work poorly under time pressure at a disadvantage.  This is good!  We often work under time pressure.

"Can the candidate correctly implement a solution to a simple problem?" is of course only a tiny part of what I'm looking for, and NOT the most important.  I'm looking for things like:

* did the candidate jump right in and start writing code without actually understanding the problem first? 

For example, I've asked people to write a class that represents dates and often before the statement of the problem is out of my mouth they're figuring out on my whiteboard how many bits you need to get nanosecond precision over a range of a thousand years starting in 1970. 

I'll usually let them roll along for a few minutes before I start to ask questions like: "Where did those numbers come from? Who said I needed nanosecond precision?  And by the way, did I mention that I need to keep track of dates starting in 6000 BC, not 1970?"

Anyone can twiddle bits -- I need to hire people who can understand user requirements. Writing code is easy.  Knowing what code to write is hard.

* does the code "work by accident"?  I've had candidates who wrote bad code, found the bugs, fixed them by randomly changing the code until it worked, and were then unable to cogently explain to me in English how the algorithm that they just wrote worked.

* Can the candidate find bugs in their own code? Can they tidy it up?  Can they optimize it for speed, for space, for readability, for portability?  Do they handle error cases?  Even if the code is _wrong_, I care about these things.

I don't really care if you can't write a correct algorithm in twenty minutes. I've recommended-for-hire candidates who did not actually correctly solve my problem, and I've no-hired candidates who did correctly solve it.  It is much more about how you approach the problem.

Anyone can write correct code, given enough time.  Not everyone can understand the user requirements, carefully analyze the problem and then apply sound software engineering practices towards a solution.  I want to hire the people who can.

Those things are way more important to me than knowing if the candidate can figure out how many piano tuners there are in Idaho.  Figuring out all that in a 50 minute hour is pretty hard, but we do our best.

Eric

Eric Lippert
Monday, May 05, 2003

I love the "how many ping pong balls" type of question.  First of all, engineers find themselves making back-of-the-envelope calculations all the time, so someone who's incapable of this lacks a very important job skill.

But secondly, it gives you a glimpse inside the person's personality.  Do they just stand there and stare at you?  If so, they likely lack the persistence to solve this and more difficult problems, and would be more fit for a career at McDonald's, only doing what they are told to do.

The "how would you move Mt. Fuji" type of question, on the other hand, is utter crap.  The only correct answer to it is violent push-back at doing such an enormous, pointless task.  The ability to contrive an imaginative solution to this problem has absolutely peanuts to do with writing real code.

"How would you design ..." questions are only marginally better.  If the item is beyond the interviewee's field of expertise, their response counts for nothing.  If you ask a field of non-accountants how they would write an accounting package, they would all be wrong, except perhaps the solitary guy who replies, "well, I would first start out by asking a lot of accountants what they want ..."

My suggestion for an interview question?  Set the candidate in front of a computer, throw them the API for a package they hadn't used before (for example, Microsoft's Crypto API), and ask them to write a simple program that requires them to figure out how it works.  Error-handling and snazzy user interface is a bonus, but far from essential.

This all said, I believe it's extremely difficult to figure out in a few hours whether a candidate would be a good fit for the company.  Many organizations have no corporate knowledge of their own employee's performance even after they've been working for them for two years ...

I'd like to see the popularity of a contingent one-month probation period for new hires.

Alyosha`
Monday, May 05, 2003

Eric,

The problem is that there is a difference between "time pressure" and "why-does-he-keep-staring-at-me pressure". When you are under the gun in your office, there is no guy standing over your shoulder watching what you do - no one ever sees and/or criticizes all the stupid little mistakes you make. Ever try to type up something (anything) when someone is watching you at your desk? People always fumble, typing and re-typing words because they're a little nervous - someone is watching them. Now add to that the fact that what the candidate is doing requires real thought, and we have a problem.

Anyone who has some interest in UI design/theory knows that stress decreases the size of a person's "mental workspace." This is best explained by Donald Norman's desk metaphor (and also by Ben Hoff in The Tao of Pooh). When something bad/scary/unexpected/weird happens, your stress level rises and your "desktop" shrinks - you have less mental "area" to work with. It's harder to keep things straight.

Having someone you just met five minutes ago and who you know is deciding your future give you a challenging coding problem and then stare at you while you try to figure it out falls into the category of Highly Stressful Events. Unfortunately, what we want to simulate is Realistic Highly Stressful Events. How many candidates have you turned down because you thought they couldn't understand the problem when, really, they were wondering if their handwriting looked okay? How many of those candidates knew The Right Way to solve your problem but freaked out because everytime they made a tiny mistake and erased it, they knew you were making a mental note about how many mistakes they had made? I would say that time pressure is not nearly as hard to overcome as the pressure of trying to look perfect in an interview.

For a company that spends so much money on usability research, you'd think that someone from Building 112 would've fired off an e-mail to Bill by now telling him his interview style needs to be tweaked.

Dan J
Monday, May 05, 2003

Hey Dan,

Those are good questions.

Here's an interesting apocryphal statistic I heard the other day. I take you now back to your high school math class:

"What's the definition of a cosine?  Bill?"

[pause]

"Anyone else know what a cosine is?  Mary?"

In the United States, the average pause time between asking the question and assuming that Bill doesn't know the answer is allegedly 1.7 seconds.  (In Japanese high schools it is allegedly in excess of 30 seconds.)  Americans are trained to answer fast or not at all, and are unnerved by long silences.

Whether that stat is true or not, I try to keep it in mind when interviewing people.  I'll let people cogitate for a good 45 seconds before I ask them to let me in on what they're thinking.

Also, I have a couch on my office.  For the coding question, once the candidate actually starts to work it out on the board, I'll go over to my couch, read over my notes so far, and give the candidate some space to think for two or three minutes without being stared at -- but of course I'm there if they have clarifying questions, etc.

I also try hard to put the candidate at ease the moment they walk into my office.  (Believe me, I've seen plenty of stress cases!)  I never jump into coding questions right away -- I get the candidate a soda and ask them how the day is going so far and where they went for lunch, and then I throw them a bunch of softball questions based on their resume to get the ball rolling.

I also know what you mean about the "mental workspace" shrinking.  The coding question I ask was specifically designed to require mental manipulation of only four numbers, all representing familiar "real world" quanties.  The problem can be solved several different ways and you can write a correct solution in one line of extremely clear, straightforward C once you suss out what the algorithm is.

Like I said, I'm not only looking to see if the candidate can write that line of code -- I'm looking to see _how_ the candidate approaches a simple problem.  I don't care if there are missing semicolons, and I tell them that ahead of time. 

But no matter what you do, interviews are going to be stressful.  I've recommended that we hire candidates who ultimately got turned down because they blew a problem in a fit of nerves in a subsequent interview.  It happens, and it is unfortunate, and I try very hard to both minimize interview stress and take it into account.

But ultimately, as the interviewer the most important thing is to _minimize bad hires_. One bad hire can make up for ten good ones.  Minimizing badness unfortunately means erring on the side of sometimes rejecting bright, competent people who interview poorly or have bad days.  Heck, _I_ was no-hired by several teams when I first interviewed, and I like to think that the interviewers just made a mistake!

So I try very hard to find and recommend the truly excellent candidates, but it does occasionally happen that good ones get away.  C'est la vie.

Eric

Eric Lippert
Monday, May 05, 2003

"I'm not only looking to see if the candidate can write that line of code"

Heh? You really care? Are you saying that if he for example forgets the order of arguments to printf you wouldn't hire him?


Tuesday, May 06, 2003

The answer to how you move Mt Fuji is to ask more questions. For example, at face value, moving Mt Fuji is silly. So, do they mean the real Mt Fuji? Or do they mean a picture of Mt Fuji.  Is "Mt Fuji" the name of their plant? If it is, pick it up and move it ... etc.

In other words, refine the requirements, THEN answer the question!! Assumptions are the MOAFU.

Gerhard
Tuesday, May 06, 2003

> Are you saying that if he for
> example forgets the order of arguments to printf you
> wouldn't hire him?

No, I'm saying the exact opposite of that.  I apologize that I was not sufficiently clear.

Of course, since printf takes a variable argument list of any type, there is no "order" to forget -- but I take your point. 

I don't care if the candidate makes silly syntactic mistakes or forgets whether sin() takes degrees or radians.  Furthermore, I tell the candidate this at the beginning of the coding question -- that they should concentrate on showing me their thought process and not worry about keeping every semicolon in place.

Eric

Eric Lippert
Tuesday, May 06, 2003

*  Recent Topics

*  Fog Creek Home