Fog Creek Software
Discussion Board




BrainBench


BrainBench Certifications are free for the next week.

(www.brainbench.com)

I took a few of them.  I found the testing and software quality exams to be particularly distressing - like, I can't _believe_ how poorly the tests were written.  That people would show off this ... crap to get a job is embarassing.

Then again, if you're out of work, and it would impress an employer, hey, the tests are free.  And they give you nifty .gif logos to put on your resume ... :-)

Thoughts?

Regards,

Matt H.
Thursday, July 08, 2004

Thanks for the heads up! :)

JD

JD
Thursday, July 08, 2004

Are there any *decent* tests that determine how skilled or effective a programmer or IT person is? There are the Microsoft certifications, but those only test how well you know Microsoft technologies, and not how well you apply them.

Then there's your standard CS degree, but again it's mostly theory and very little (if any) application in an engineering environment.

Derek
Thursday, July 08, 2004

Hm...the only tests I can think of that really test how well you apply your knowledge are the ones in the school of hard knocks.

Does the app work?

Aaron F Stanton
Thursday, July 08, 2004

The problem with having tests to get qualifications, is that first you need to get agreement on what knowledge is needed -- which we in the software engineering / computer science world do not have.  Yet.

It's coming.  Until that agreement is developed, all you can do is test small domains of knowledge for limited purposes.  MCSE exams come to mind, and Cisco certifications.  Areas where the needed skills are reasonably well-defined, and questions can more-or-less determine if you have those skills.

AllanL5
Thursday, July 08, 2004

Derek writes: "Are there any *decent* tests that determine how skilled or effective a programmer or IT person is? "

The Sun Certified Java Developer test is an actual project, based on an intentionally somewhat vague requirements document. There are a number of design and coding errors you can make which trigger automatic failure. (For example, writing code which can result in deadlocked threads may be grounds for automatic failure.)

A typical assignment has a GUI portion, a networking portion, and some back-end-y stuff, such as doing things with files.

You're required to document the design decisions you make (in response to the vague requirements, for example) and write a user help document for the program.

The assignment gives you some leeway in how you design the software. For example, you might be given the choice of using sockets or RMI to implement a networking function. Design restrictions often are intended to simplify human and computer grading. They don't want designs with lots of unusual installation requirements or configuration quirks.

When the test-taker completes the project, she submits it to Sun over the net as a jar file. The work is evaluated by a person, who uses some automated tools to test the code and presumably their brain to evaluate other things.

After submitting their project, the test-taker goes to a test center and takes a computer-based essay test which asks questions about the project and the decisions they made. I expect this is to catch people who didn't do their own programming.

This seems like a pretty good, challenging test, assuming the test-taker doesn't cheat and the grading is tough. The person has a year to complete the assignment; the work is not done at a testing company location.

The pre-requisite, the Sun Certified Java Programmer test, is an easy multiple-choice test, which I don't think says much about people who pass it.

Jon Hendry
Thursday, July 08, 2004

> I  found the testing and software quality exams to be particularly distressing - like, I can't _believe_ how poorly the tests were written.

Which tests did you take Matt? I just took the C exam and found it to be rather challenging -- there were several questions that confounded me. Passed but didn't do as well as I'd like to have done.

Tony Chang
Thursday, July 08, 2004

Here's my Brainbench story:

About 9 months ago I decided to look for another job.  My employer at the time did not use .NET, so I couldn't upskill in it at work.  I spent a lot of time learning about .NET in my own time, but found that employers were reluctant to employ someone without "commercial .NET experience".  My significant, and successful, experience in _other_ languages didn't seem to count.

I looked at Microsoft's .NET certifications but I couldn't afford them. 

So, I took the Brainbench .NET Framework test.  As I worked though the questions, I found it quite difficult.  What I didn't take into account was this: as you go through the test, the server chooses each new question based on how well you've done so far. So, in theory, you should always find it hard, because if you're doing well it will just ask harder questions.

Not realising this, I waited for the final results screen with some trepidation.  "Congratualations", read the screen, "you have scored higher than 98% of all previous test takers, and appear capable of mentoring others in most aspects of .NET development!".

Well, after I finished laughing, I realised that the 98% figure would probably be helpful in getting a job.  (It was.  While several companies still turned me down anyway, I finally came to one where the interviewer started to question me about my "lack of experience". He glanced back at my CV, mentioned the Brainbench result with words to the effect of, "Well I guess that answers _that_ question", and moved on the the next question!  I got the job.)

On a less positive note, I felt that the comment about being able to mentor others in most aspects of .NET development was a bit over the top, and reduced the credibility of the tests.  How could that be true for someone who had done nothing but study .NET in evenings and weekends! Even I didn't believe that.  However, after a short time in my new job I _was_ mentoring others, even those with "commercial .NET experience".  So, perhaps all that study really did pay off, and the test was accurate afterall.

I know this sounds like some kind of informercial, but its not.  It's what happened to me and you mileage, of course, may vary.

I'll omit my name because, in spite of how things have turned out, my employer is still afraid of their client finding out that I have "no experience."

Deliberatey Anonymous
Friday, July 09, 2004

Interesting, I nipped off and took the C one. Fairly reasonable except that some of the questions were a bit too much "heres lots of pointers with consts" which in the real world you'ld fix with some typedefs. Bearing in mind that I haven't written any C for over 6 months, spend most of time in VB/C# and const was invented after I started with C[1] (and I've never really "got" it) I managed to get a score of 4.08 which it said was in the top 95%.
I suspect if I wanted to really pass such a test then I'd do a bit more preparation (i.e. reread up on some of the pointers to pointers definitions without typedefs) and get a much better score.
Fun though. I might have a slightly more serious go at some of the C# / .NET ones more for my own satisfaction than anything else.

[1] I started with C in 1984 which was pre-ansi and prototypes were a brand new idea.

Peter Ibbotson
Friday, July 09, 2004

If you can score pretty high on TopCoder contests, you're probably at least a badass at sorting and other fundamental algorithms.  Of course, if you can do that, you'd make enough money from the contests to not really need a job necessarily.

Personally, I found the brainbench tests to be pretty useful.  Most biz's want a certification in some piece of software, but they really don't specify which cert.  So, you get a brainbench one because they cost the least, and since all cert's are essentially worthless in the sense that the cert study materials would be available to you at any job, then you can apply any material necessary on the spot and be fine.

What bothers me the most is that a person would need to get certified in a piece of software in order to learn it, because the manuals are crap, and only the cert study guides actually go over how to do anything.

sir_flexalot
Friday, July 09, 2004

TopCoder is a brilliant website...

They take the work you write in the compititions to solve business problems for their clients.  They pay you some scratch, and they get paid by their clients to implement it...

Charley Bogwill
Friday, July 09, 2004

So the OP says hey look "free for the next week" and then he says it's crap. Oh. Like saying the hotel is free but only next month and watch out the bed bugs bite. What's the point? I wonder if it's just someone doing marketing for BB. I used them. I found them to be morons. They wanted me to prove who I was by faxing them my driver's license. Wow, what security. I refused and so I got certs with a fake name I used to register as I didn't want their spam.

About the tests. It doesn't matter how easy or hard it it is. The interesting thing is to see how you compare with the other people taking the test. I found it useful for this.

Also, I was like the other guy above. I studied PHP for a few months and I scored higher on it than I did on Perl which I'd been using for a few years. BTW, at a head hunter I scored higher on PHP than Perl and I even scored well enough to be sent as a sysadmin which was a laugh. 

But about tests for programmers. Would you give a grammar test to a writer? A spelling test? Would a writer have to type X wps or his handwriting be perfect? Not my writer. He'd have to be able to communicate with the written word. I'd hire someone else to check grammar and spelling.

But for programmers.  Oh no, I would not read the code they had written. What would that tell me? Heck, could I even comprehend it? No, I just need someone else to tell me if this person can take the test or not. Oh please.  Is it any wonder we have a shortage of skilled workers when seem to have a surplus of morons doing the hiring.

BTW, I never got a job owing to BrainBench but I still found it useful as a measurment.  Also, you can have great fun with the people who pass all the tests with flying colors and spend time on their boards complaining about not being able to get jobs.

me
Friday, July 09, 2004

*  Recent Topics

*  Fog Creek Home