Fog Creek Software
Discussion Board

Reality check on programming 'test'

A friend emailed me a solicitation that he found on a work-from-home board for a remote software contractor. His opinion was that the guy was a jerk/control freak. However, the technical requirements had "me" written all over them.

So I emailed the guy and got an auto response. The "test" follows at the bottom. It is basically a request to implement a small internet aware app in Delphi and to implement a PHP script on the server to provide the web service.

I feel this task is brain dead simple. It is a legitimate request for a mix of certain skills. It is probably 2-4 hours of work at very most. And it doesn't 'feel' useful enough to be worth conning people into writing free code. I am considering doing it. I've wasted entire days interviewing with stupid asses and getting no where. In terms of obtrusiveness, this isn't really bad.

Is this a legitimate recruitment tactic? Is this something that other SW companies should consider doing? Or do you see it as demeaning, insulting, etc?

You must write a simple server/client application in Delphi/PHP.  Delphi
will communicate with PHP using the WinInet library, by making HTTP
requests.  The delphi-side code will consist of a component which keeps
synchronized time with the remote server running PHP, without changing
the local time of the machine it is running on.  It only needs to
support one property (other than the ones that make it work),
TMyComponent.ServerTime, which returns the number of seconds since the
Unix epoch, equivalent to the time on the remote server.  Time should be
accurate 100% of the time, give or take a few seconds max, to allow for
network latency.  Some challenges to keep in mind that come with this

1.  As stated above, the Delphi program cannot change the local clock,
so it must keep track of time another way.

2.  If you rely on the local clock to do your calculations, keep in mind
that the user could change the local clock at any time - this cannot
lead to the wrong answer eturned by ServerTime

3.  Threads should be involved in some way

4.  The code must be written purely by yourself - using no third party

5.  The code must be commented, documented, and of course, bug free.

Business type wanting to debate
Saturday, June 12, 2004

It's a market. If you want the job you will do the
test. I know that's sounds simplistic, but from the
other person's perspective he probably has been
ripped off so wants very tangible evidence of skills.
As it is unlikely this problem is a commerical
project he is not trying to get something
for free. Of course, you may want to test back
to see if he will pay and how he responds to
questions etc.

Though, if i had other options, i wouldn't do it.

son of parnas
Saturday, June 12, 2004

>3.  Threads should be involved in some way


Anyway, you're right about it being brain-dead simple.  If that's representative of the kind of work you'd do for that company, I'd skip it and let some students take the job.

Saturday, June 12, 2004

Without commenting on that delivery scheme, or the particular test, I'll tell you that when evaluating a programmer there is simply no substitute for looking at his or her code.  None.  And it must be looked at by at least one very, very, very good programmer, which itself causes a nearly intractable problem for 95% of the shops out there, who don't already have one.

An article I stumbled onto this week proposes and interesting idea about that last problem of having nobody qualified to qualify the hire.

That testing and thorough evaluation of talent is not the common practice in the IT industry, and to some extend the software industry, is alone sufficient to explain nearly every problem in the world of software.  (Although sufficient, there is an abundance of problems lined up behind that one.)  It's remarkably easy for a talentless amateur to pull on a suit and convince gullible know-nothings that he's a programmer by virtue of having done the same for 10 or 20 years running.

Without having a top programmer look at the candidate's code (and, of course, talk to the candidate for at least a few hours too!) the best any hiring company can hope for is a 1 in 10,000 chance of ending up with a top performer rather than the typical 2nd or 3rd rate time-gobbler.

Saturday, June 12, 2004

Oh, and if you're hiring process is not rigorous, the best programmers can smell the mediocrity, and usually won't take your offer anyway, unless the market is such a shambles they just want a place to camp for a few months.  So your already miniscule odds of getting better than a seat-warmer drop even lower.

Saturday, June 12, 2004

The funny thing about that test, is that, given those set of requirements, my first questions would be:

* How often is the property going to be checked?

* How quickly does the property need to return a result?

* How often may I query the server?

Because depending on the answers, I might just have any querying of the property simply go ahead and ping the server at that point.  Thus, the property will be correct "100%" of the time that it is checked, plus/minus the latency.  No threading required, and the "simplest thing that could possibly work".  :)

Actually, for that matter, there's no reason to use threading anyway, even if you only ping the server periodically.  I would simply use the Windows API 'QueryPerformanceCounter()' and 'QueryPerformanceFrequency()' functions to compute the current value of the property, and use standard Windows timer callbacks to fire the HTTP request.

So, I'd also bring that up with the person as well.  That is, why did you specify threads when it's not necessary?

Oh wait, I suppose this WinInet library must be synchronous, and not allow you to say, receive an event, message, or callback with the result of your HTTP request.  In that case, you'd need a worker thread to make the requests and tweak your time offset.  Ah well.  Probably best not to expand the specs to adding async HTTP routines, if the requirements are okay with using threads.  :)  I just hate to use threads unnecessarily, as they're usually more trouble than they're worth.

Phillip J. Eby
Saturday, June 12, 2004

The reply that best matches my feeling on this is:

>> It's a market. If you want the job you will do the
test. I know that's sounds simplistic, but from the
other person's perspective he probably has been
ripped off so wants very tangible evidence of skills.
As it is unlikely this problem is a commerical
project he is not trying to get something
for free.

I mentally put myself in the hirer's shoes and I agree with this. The test does several things. The candidate has to be motivated to do the task. The candidate has to write new code that will be examined by peers. The program has to do something quite specific.

It's pretty definitive. A lot of wanna bes can handwave while having absolutely no ability to produce to a result. I actually think it's not the worst way that the hiring party could go. Maybe not the best, but not bad.

The other comment on this thread that "using threads in some way" elicits a "WTF?" - my guess is that the hirer wants to test the candidate's ability to use a conceptual feature in some way. IE: in this application, a thread could implement a background polling loop that periodically obtains the server time.

Of course, to get back to the problems of wanna bes, a wanna be/poseur could SAY in a confident sounding way "write a background thread that polls the server" while not having a whit of an idea how to: create a thread; make a thread execute; use wininet to poll something remote; communicate the result to a foreground thread.

On one hand the premise that "everyone who applies to my job is probably a G** D*** liar" tends to guide this type of exercize, there are a LOT of poseurs in our field who have tech writer type skills to talk intelligibly while not being able to carry even a simple task off, or turning it into a boondoggle.

Also, as mentioned in the past here, the programming field has the phenomenon of "feasting". IE,  people who are well employed - not really needing or wanting to make a change - spamming out resumes just as a competitive thing. A lot of programmers are frivolous time wasters.

>> Though, if i had other options, i wouldn't do it.

Agreed. That's another litmus test. If the candidate can't spare the time, then either they don't have the time to spend on a contract job, or they don't have an attitude that the hirer deems acceptable.

Business type wanting to debate
Saturday, June 12, 2004

Phillip Eby, I agree with all that you say.

My guess is that the hirer just wants to see what the person does with the requirements as is.

Maybe the omission of several important parameters is a test of the candidates' ability to improvise with a reasonable solution.

Yeah, threads are overkill here. As I stated, the hirer probably wants to see if the candidate knows what a thread *is*. Even if the requirement driving it is artificial.

Business type wanting to debate
Saturday, June 12, 2004

Does passing this test guarantee an interview?  If not, it will only attract unemployed and desperate people.  Good programmers who are already employed aren't going to want to waste their time if there is no assurance that there is a decent chance of benefitting from it.

T. Norman
Saturday, June 12, 2004

>> >>>>>>>>>>
"Does passing this test guarantee an interview?  If not, it will only attract unemployed and desperate people.  Good programmers who are already employed aren't going to want to waste their time if there is no assurance that there is a decent chance of benefitting from it."

I certainly hope you're not saying or even implying that unemployed programmers are bad or that only "good" programmers are currently employed.  This is totally false and I am extremely offended that someone would even make a statement that implies such a thing.

Saturday, June 12, 2004

There are good and bad employed programmers, and there are good and bad unemployed programmers.

But the good ones, employed or not, will be less inclined to jump through hoops before getting an interview.  So a test like this that has a high probability of being a waste of time will produce an applicant pool that is disproportionately full of candidates who have no other options.

T. Norman
Saturday, June 12, 2004

Plenty of software companies have full-day interviews. I fail to see how this is different.

Rhys Keepence
Saturday, June 12, 2004

It's different in that the software company has invested a day of their staff's time to the process of matching applicants with jobs. They are investing as much in the process as the candidate.

In the case of a pissy little paper test, some fly-by-night merchant is requiring the candidate to expend much greater effort on the process than the alleged employer is.

Also, in this case, it looks suspiciosly like the product of the so-called test will have some value to the alleged employer. This is fraud.

In general terms, has programming really sunk so low that idiots would fall for this crap? What would an accountant say if you asked him  to audit your books for free, as a test?

Inside Job
Saturday, June 12, 2004

A full day interview is an actual interview.  When you get to that stage, you know your resume hasn't been shredded, and there aren't 100+ other people still in the running for the position.

That's why I asked about whether passing this test would guarantee an interview (given that your resume meets the stated job requirements, of course).  It would be acceptable if they only gave the test to people who they already decided to interview, or if the test is good enough as a screening tool that they could and would interview everybody who passed the test.

But to spend hours when you don't even know if the position is not fake and you don't know if HR has or will trash your resume, is a lot of effort for little chance of reward, and good candidates with other options will be deterred from it.

T. Norman
Saturday, June 12, 2004

It's also different in that a full-day interview (hopefully) will test more substantive aspects of a developer's ability.  Namely, the ability to function in a team and develop software in a business environment.

Honestly, there are two extremes of "undesirables" when it comes to applicants.  On the one hand, there are the aforementioned "poseurs" who don't know jack, but who can talk their way into anything.  But on the other are the "techno-weenies"; the prima-donnas who thrive on individual assignments, but who can't function in a team/business environment.

So any substantive (measured in hours) time spent in a job applications process should try to find people who are on neither extreme.  That's why real, full-day interviews involve having people meet members of the development team and management, in addition to showcasing ("proving") technical ability.

Pre-screening is just that:  screening.  You really have to put effort ($$$) into finding good applicants; to me, this test is just the employer naively hoping that if he increases the time burden of the pre-screening, then magically great applicants will come forth.

Unfortunately the "soft skills" of good candidates are disproprotionately more important than the mere ability to write a "dead simple" application.

Sunday, June 13, 2004

Alright, I don't know delphi, but suppose of a poseur, what would prevent me from simply looking for the code on the net, sewing a few pieces together and wa-lah!!!

the artist formerly known as prince
Sunday, June 13, 2004

You know, here's an interesting idea...

if I was trying to get a project done for cheap (FREE), I'd split it into little pieces like this (maybe use case-based?) and farm it out to "applicants".


Sunday, June 13, 2004

Communist China tried that with the Backyard Steel campaign. The idea was to get everyone in China producing a small amount of steel, and presto China is able to produce a large amount of steel. The problem came when varying quality of the steel made it impossible to use together. No doubt you'll have the same problem when trying to put all the different modules together.

Matthew Lock
Sunday, June 13, 2004

I like this as a test. The threading bit is a clue that async requests are required etc. it should also show wannabees because they'll miss all the locking stuff thats required in the real world.
It's simple to write, slightly ambigous in places but not where it matters and hopefully good applicants recognise it for what it is, a chance to show that you can do async tcp stuff.
It's also sufficently unlikely to be doable with code found on the web, and useless enough not to be part of a larger project.

Peter Ibbotson
Monday, June 14, 2004

I've seen this up here in Toronto. I received this test from a recruiter:

I was told that completing the test practically guaranteed an interview. I decided not to do the test personally, but at least there's a reasonable chance that the time invested would have a 'payoff' for someone wanting that particular job.

I like tests like this for entry positions, with this caveat: if you administer the test, interview *everyone* who completes it properly, even if they don't have the right experience or degree.

Monday, June 14, 2004

Tests like this are proof that as much as we may like to think of what we do as a "profession", it sure as hell isn't.

The limited (almost zero) barriers to entry into programming mitigate against it ever being a profession.

Generally in other professional careers, you need a degree and/or a (hard-to-get) professional certification to be allowed to practice - in IT you don't.

Chris Welsh
Tuesday, June 15, 2004

Oh what a mish mash. I love this last guy. Friend, I'll tell you, the dumbest people I know are the collegiate graduates. Some very smart people I know (very few)  did college but I don't judge someone by what they did in school ... I judge by what they've done recently. Actually, I guess this is hard for other people, I can watch someone work for a short while, I ask them what tools they use,  ... I got em.  Personally I've found a lot of good people who came through the military.  Then there are people who have a passion and it wasn't because they were too dumb to be a doctor or a lawyer.

And gosh ... for the first time I see somebody mention reading code. But then he talks about needing to have someone very,very, like super very good read the code. He falls into the delusion that there are enough Kobe Bryants to go around. 

A company will need some very strong people to be mentors to the rest .... some of those rest will become very strong ... maybe not in the same way/language of their mentors and some of the rest you should let go ... some will go on to become mentors at other companies.

But a test like the OP talked about. I don't know Delphi. Personally nothing is ever easy to me. I avoid easy like the plague. I want to to know context, framework, goals, the future. If someone wants to see how I write code I'll show them my code but I don't do tests.  I hope someone wouldn't test Shaq by his free throws but it makes for a simple test eh.

Tuesday, June 15, 2004

"If I was implementing this system for real I would first ask blah blah and blah, on which my solution would depend by blah and blah. This requirement seems odd, I would prefer to do it blah way. Would you like to see such a solution? I have created the code as specifically required, attatched."

Is there any way to include a deliberate error to make the code worthless commercially? Without looking like an actual error...

You could also send a message with a throw-away email address and see if you get the SAME challenge every time...

Jack V.
Tuesday, June 15, 2004

My point had nothing to do with ability - it was about a "profession". Without barriers to entry, IT is and cannot be a profession.

And as an aside, round here where we have universities, I've found very few dumb graduates. Plenty of smart people without degrees, mind you, but that's not the same thing.

Chris Welsh
Tuesday, June 15, 2004

To Chris:

The thing with programming and schools is that programmers have traditionally always been self learners. Schools have followed *afterwards*, as the programmers themselves have created the software industry. And even today universities still produce mostly very mediocre programmers.

To me, being a self learner and early adopter seems more valuable than the ability to pick something fed to you in school anyways. Schools are just a tool for learning a skill, and if you can learn it by yourself, you don't need the school. Except maybe for the real world value of the degree to convince old fashioned twats that you're 'eligible' for the job.

That's why programming positions won't require a degree until that degree really brings into table something that the person couldn't easily learn by himself (and even then if he did, he's proven to be smarter than those who HAD to go to school to learn it).

To topic:

I don't see what's bad about doing that test. If the person uses that code commercially, who cares, it took you a few hours, and it's not like you just gave away the best thing since sliced bread anyways. If you really want the job, you should do the test. If for no other purpose than to see what kind of salary they would offer you.

Friday, June 18, 2004

Coincidentally, I found this article after writing my previous reply, I think it's quite truthful:

Friday, June 18, 2004

*  Recent Topics

*  Fog Creek Home