Fog Creek Software
Discussion Board




How to hire for data conversion

I'm helping out a friend from a previous employer who came to me with a job offer to do some data cleaning and conversion work, which I have done several times in the past as part of larger projects.  This time though it's nothing but 9 months of data conversion.  It seemed like such an icky and thankless job that I turned it down, but offered to help find someone to fill it.

My main problem with data conversion work is that it's primarily about cleaning up someone else's mess and trying to get them to help you do a good job of it.  You're a high-tech janitor, and once that's done you're the plumber that pipes it from one place to another.  Nobody appreciates all the hair-pulling data quality issues you had to deal with.  They expect you to magically "make it fit" one way or another, even when that is impossible.  Explanations may fall on deaf ears, since issues often involve obscure details of the data.  Only the flaws in your work are visible.

Another issue is that it can be quite difficult to define what "done" means, since it's all about achieving a fuzzily-defined level of data quality.  There's a lot of room for the employer to insist that your definition of quality is off.

Where do you find someone that actually enjoys this kind of work?  How do you sell the position?  How can I change my perspective to see this job in a better light?

ODN
Monday, June 02, 2003

Lets say their ready to pay you $10.00/hr for 9 months to do the data conversion. (convenient number)

  $10.00      40hrs    4weeks    9mon
  --------  X  -------  X  ---------  X  -------  = $14,400
      hr          week      mon

Lets say you offer to do it in 6 months for $10,000

So you write a conversion spec. listing exactly what must be done.

You post on the internet that you have piecework available where people can make money from home and you make them sign an agreement on acceptable errors.

You farm them out, consider a profit margin of about 40% and cost out the price per conversion. Then, take a vacation for the next 3 months.

Hows that sound?

Kent Design4Effect
Monday, June 02, 2003

at my previous place of work they took this kind of thing (integrating legacy systems) VERY seriously. It was a multi million dollar project.

I agree with the previous poster... have you and your friend farm it out to India and live off the profits.

www.MarkTAW.com
Monday, June 02, 2003

>> "You farm them out, consider a profit margin of about 40% and cost out the price per conversion. Then, take a vacation for the next 3 months."

Sounds like a way to ruin your reputation.


Monday, June 02, 2003

Your comments Kent remind me of the old Syrian joke about Nasser and the astronauts.

The story goes that Nasser was green with envy because the Americans had sent a man to the moon before the Arabs, and he was going to show the world what a Socialist Arab Republic could do.

So he got his scientists to build the rocket and then advertised for the astronaut.

The first applicant was Russian. "How much do you want?" Nasser asked. "Well," the Russian said, "It is a risky job, and you've never sent a rocket up before, so I'll have to have a million dollars."

The next applicant was an American. He wasn't exactly keen on going up in the thing, but he had just had an expensive divorce, so what the heck! "Two million dollars" he said.

Then Nasser asked the Syrian astronaut how much he wanted. "Three million dollars," was the reply.

Nasser was furious."The Russian has been up in space loads of times and only wants a million dollars, and the American has been to the moon but still only asks for two million, yet you haven't even been up in an airplane and you're asking three million. Explain that!"

"Simple," the Syrian astronaut said. "A million for me, a million for you, and a million for the Russian!"

Stephen Jones
Monday, June 02, 2003

You need someone who has done this before.  While you may be able to do it, you are telling someone else, that you know someone else can. 

Conversions are expensive because they have to be correct, as you point out.  What is acceptable and how much it will cost vary greatly.  Get the specs out and ask for an RFP from a few companies.  You stand a better chance of success and less risk.

Unless you have a relationship with an off-shore company, I would avoid them for now.  While this work is right for that type of work, it can also be a great headache if it does anything but work perfectly.

Good luck.

Mike Gamerland
Monday, June 02, 2003

I hope nobody thought I was serious when I said farm it out to India. I mean, this is obviously a job for the dislocated American in Syria.

www.MarkTAW.com
Monday, June 02, 2003

ODN - if you do end up involved in the contracting process at all, be sure that there's a percentage accuracy in the contract - 99.5% or 99.999% or whatever you negotiate (obviously the more 9's, the higher the contract price)

If that number isn't in there, then the client can argue for 100% accuracy, and if the data is "fuzzy" then it may be impossible for a contractor to achieve 100% accuracy.

Also be sure to indicate in the contract that the migrated data will be faithful to *the original data*, not the ground truth. (GIGO)

Philo

Philo
Monday, June 02, 2003

Hire someone who can think. Seriosly though, I would never attempt such a job without someone who can think. Existing data will need to be understood. Descisions on the best way to convert will need to be made.

Throw in someone who knows data modelling. Not just someone who is familiar with SQL, but someone who gets turned on by data models! How far to normalize. What data types. Where indexes, views, etc?

Depending on the volumes of data, someone familiar with a scripting language of sorts (perl) to automate the process.

This really is a thinking job.

Whatever you do, do not entrust your data migration to a Data Entry Clerk. Best case scenario, they don't complete the job. Worst case, they corrupt your data.

I would compare data migration/conversion to refactoring. If you are going to refactor code, you probably want your sharpest man doing it, and not some cut-and-paste programmer doing it.

tapiwa
Tuesday, June 03, 2003

> I would compare data migration/conversion to
> refactoring. If you are going to refactor code, you
> probably want your sharpest man doing it, and not some
> cut-and-paste programmer doing it.

Good analogy.

www.MarkTAW.com
Wednesday, June 04, 2003

> I would compare data migration/conversion to
> refactoring. If you are going to refactor code, you
> probably want your sharpest man doing it, and not some
> cut-and-paste programmer doing it.

The problem with this, as the original poster is obviously in this position, is that once

So effectively the 'sharpest man' is not likely to find the job all that attractive.

As explained, if you want to get things right you need to do a lot of questioning and explaining which noone else appreciates. Last time I did something like this was for a client relationship management system and the users of the old system had given the data 'extra meaning' (e.g. Adding 100 years to a specific date meant y instead of the usual x etc. etc.) Tracking down and resolving these issue's weren't much fun, and not at all satisfying.

It was a good feeling knowing things were correct once I'd sorted it all out, but I don't think I could handle 9 months of it, and assume the most suitable people would be turned off it for the same reason...

Gordon Hartley
Wednesday, June 04, 2003

Hi, I have developed a Data Conversion QC Software which will save your 50% work time. It is available at very affrodable cost. You can email me with your details for free 40 times trial version of this software.
subratambharati@rediffmail.com

Subratam Bharati
Saturday, June 26, 2004

Dear Sir,

Thanks for your mail. Please provide me some work (Data Conversion / Typing Job from home through Internet).

Thanks once again for your kind support & patronage.

Masud Chowdhury

Masud Chowdhury
Saturday, July 03, 2004

*  Recent Topics

*  Fog Creek Home