Fog Creek Software
Discussion Board




Offshore development realities


The big buzz the past year has been offshore development.  Just curious what the experiences/information of the JOS community is on this?

I know this is becoming mainstream at least in upper level management meetings as long term strategic to cut IT costs.  Case in point a friend who is a director of operations of a large bank software provider says they prototyping outsourcing develop offshore to India for a couple of projects to see how it works in the next year.

I know of at least two projects where outsourcing to India was an utter failure.  Quality of product was terrible and of course there was no system documentation.

Anyways interested to hear more stories...

/James

James Nicoll
Wednesday, June 18, 2003

I was peripherally involved in a project that a vendor took care of completely. We gave them our requirements and they returned a product. It was always what we asked for, and it was always on time. Perhaps some of the design philosphy was questionable, but they were probably jumping through hoops to meet our stupid requirements.

They were fired and developers were shipped in from India. From what I could tell it was partly financial, and partly so we could be more flexible in meeting the business needs. Part of an overall push to reduce cycle time from months to weeks or something.

I have no idea what it was like at the vendor, but I know what it was like with the offshore developers we imported.

In a nutshell, they were managed poorly. The development manager was disagreeable, so nobody ever communicated with him - talking to him usually meant being asked to stay the weekend or something like that. Not many of the developers really understood the requirements and just coded to the static mockup.

The manager was also disagreeable, and new to this project so he had to prove to the upper mgt. that he could jump through hurdles and deliver anything they asked for in any time frame they asked for (and they loved him for it, people who are otherwise hip to idiot managers liked that he said he could get things done). So he cracked the whip.

I saw more than one person literally peek from behind their cubicle wall and RUN LIKE HELL down the hall with their posessions in their hands.

I was let go before the project rolled out, but I suspect it rolled out on time or near on time with a few bugs and was quickly cleaned up. I hear the manager has taken to leaving at 5 or earlier now that he's proven himself and stopped cracking the whip, but the development staff still eats an early breakfast and late dinner at the office.

And they get paid around $40 a day.

Anonymous Coward
Wednesday, June 18, 2003

I 'spose it depends on the project, but at my place of work we do development to support my company of about 350 folks (worldwide) and our clients.  With our projects we have development problems related to requirements and we share the same building with product development and the users.  I can't see outsourcing any of our software development.

I suppose if you were a big enough company and you had GREAT requirements which rarely required changes you could be successfull with outsourcing, but I don't think that applies to the majority of development work.

chris
Wednesday, June 18, 2003

So, basically, upper management got what they wanted - more flexibility, less expensive, and closer to the people who were asking for the work to get done.

On the inside, you had to feel a bit like Philo in the Camel project. In fact, I had actually wanted to start a web log while I was there about the business practices, but never ended up doing it.

Anonymous Coward
Wednesday, June 18, 2003

The problem I see with outsourcing  becoming a really big thing is that most companies appear to be still "prototyping". I suspect the prototyping will go well and they will all start rolling out round about the same time, and it will be then that they find out that the supply of qualified people is limited.

Intelligent companies will just up wages to keep the best people as there is a hell of a lot of profit margin, but others will start hiring warm bodies and things will be messy.

The habit of bringing over Indian programmers and paying them Indian wages has taken off in the last year or so. Unlike H1B's however there is common cause between the exploited Indian programmmers and the unemployed US ones, so I suggest some people start writing to their congressmen insisting that those bought over as temporary placements are only granted visas if they are paid the US rate for the job.

Stephen Jones
Wednesday, June 18, 2003

> so I suggest some people start writing to their congressmen insisting that those bought over as temporary placements are only granted visas if they are paid the US rate for the job <

That would defeat the purpose of whatever free trade agreement that allows them to come over in the first place.

www.MarkTAW.com
Wednesday, June 18, 2003

Nope. You might run foul of the WTO if you failed to let them in in the first place but there is plenty of precedent for insisting that a foreign companies workers are paid the rate of the host country. In the UK the big case was that of a load of stonemasons who had been bought over to build a Hindu temple and slept inside the half finished building for salaries of $40 a week.

Remember you're going to have both sets of workers, and the unions and governments in both countries on your side on this one.

Stephen Jones
Wednesday, June 18, 2003

In fact there is no "WTO agreement" for this type of transfer of cheap workers.

The Indian firms are notable for the extent to which they have deliberately abused visa processes in every country they operate in, in order to bring in large numbers of Indian programmers.

Now that governments and native workers are starting to clamp down on these abuses, the Indian companies are actually trying to get a new WTO agreement that removes all immigration restrictions. Not just for programmers either.

analyst
Wednesday, June 18, 2003

And just to add some fact to all the H1B fiction: The company bringing in the developers actually needs to document their salary and pay a rate comparable to a 'native' programmer - otherwise the H1B does not get approved.

Which is why many companies don't use H1B - instead, they open a subsidiary in India and declare it part of the employment exchange program. (J visa? O visa? Can't remember the letter)

THAT does not have the above requirement, and they keep paying indian salaries. Added benefits: Programmers can hardly afford a decent living space and thus spend more time at their company.

Groby
Wednesday, June 18, 2003

And we go back down the route of the classic H1B visa discussion...

Flamebait Sr.
Wednesday, June 18, 2003

Our company outsources half of our work to a team of Eastern European programmers.

Their code is marvelously designed (better than ours), robust (better than ours), on time (more than ours). The only thing lacking is code-level documentation because of the language barrier. However, the code itself is so elegant and simple to read that it is really self documenting.

And of course, we pay them diddley squat compared to our own programmers.

I myself am an H1B (and have previously held TN's, L1's, etc..). Times are changing, pay equity will come, you can't avoid it.

shabp
Wednesday, June 18, 2003

My experience has been companies are using it for new development and leaving their own people to maintain the legacy systems.  As the new systems come on line, some of the legacy people are retained, but only as subject matter experts.  The rest are being let go. 

While I have heard a lot about how it cannot be done with "my" company.  The facts are that the cost is about 12% of a US developer.  That means they can be real screw-ups and still accomplish the task for less.  However, they are not screw-ups.

Where I see this failing is the first time a third tier or even a second tier country goes to war.  Imagine the impact three years from now, as pointed out in another thread, if India goes to war with Pakistan.  They now make up 60% of your development team. 

Could the US go to war?  Certainly, however we may have greater control over it, or at least may see it coming.

Mike Gamerland
Wednesday, June 18, 2003

- continued -

Even with the risk, we will all need to use them.  A recent contact I bid for a state in the US I was under bid by a lot.  After reviewing the winning bid, I found the winner was using off-shore resources at $297/day total.  All very legal. With States in a financial crunch, $$$ count.  As business sees the savings they go to.

Do I like it ? - No  But, I like unemployment and bankruptcy less. So that leave me in the next bid looking for inexpensive resources.

Mike Gamerland
Wednesday, June 18, 2003

>"The facts are that the cost is about 12% of a US developer."

An Indian salary might be 12%, but office real estate is not so cheap ... the square footage the programmer occupies often costs more than his/her salary.  Computer hardware (including servers, routers, etc) and Internet access are usually *more* expensive than in the US.  Americans barely blink an eye at the price of a PC, but that $2000 PC in India is likely to cost more than 1/3 of the developer's salary.

So even without accounting for the added time and money costs of communication, you're already spending a total of at least 25% of what you'd be spending on a US developer, not 12%.

You might think that at 12% you can afford to have screwups doing work for you, but even a screwup working for FREE is very costly.

This wave of outsourcing is another bandwagon just like the dotcom boom. It is amazing how sheeplike American managers are. When others were spending gobs of money on the Internet, they had to follow.  When others started laying off masses, they had to follow.  When others started offshoring, they had to follow.

Corporations are diving into offshore outsourcing thinking it is an automatic thing that will save them money, just like they thought the Internet was an automatic money making machine.  Eventually they will realize that the only ones who really save money are those that have the particular managers, processes and projects that are well suited to outsourcing. Then they will scale back just like they backed off from trying to sell everything under the sun via the Internet.  But it will probably take a few years, a few billion$ wasted and thousands of more jobs losses before they come to that realization.

T. Norman
Wednesday, June 18, 2003

A couple of years ago I was involved with a project from a USA based company that farmed all their hardware and software development work to a company based in one of the former USSR Eastern Europe states.  I am based in the USA and made a couple of trips over there.

Came away very, very impressed with the sheer level and quality of work those guys cranked out.  Their work ethic was without peer - these guys just worked like bees, started early and finished late.  They seemed highly intelligent and educated, and most of the people in the office spoke at least three languages - including the receptionist.

I'd hire them anytime.  Their total cost of doing bidness was about half of the cost of doing it over here from what I could gather from the USA company's brass.

Mitch & Murray (from downtown)
Wednesday, June 18, 2003


> The company bringing in the developers actually needs to document their salary and pay a rate comparable to a 'native' programmer - otherwise the H1B does not get approved.

Actually, this is not correct. The determination of the "prevailing wage" is left to the sponsoring company, and easily undermined. Immigration lawyers boast about the ease of defeating this requirement.

The other type of visa now being systematically abused in the L-1, which was intended for senior managers being transferred to international branch offices. Indian companies now uses 1,000's of these within a single company to transfer programmers with 6-months' experience.

Nothing is inevitable.

analyst
Wednesday, June 18, 2003

I have heard good things about the quality of Russian and Eastern European developers.  But there aren't masses of them like in India, and the language barrier presents a problem if they are going to work on the in-house systems of large corporations.

Give them something technically challenging like voice recognition or device drivers and they'll do it great, but don't expect to save money if you need to get them to understand and program the business rules and regulations involved in a group disability insurance policy.

T. Norman
Wednesday, June 18, 2003

I'd say since only 26% of all software projects succeed (http://www.dotnetmaailma.com/dotnetmaailma/seminaarit/online/EMEA+Architects+Tour.htm Day 1 Session 4 )regardless of wheter they are outsourced or not you are going to have an easy time finding outsourcing failures.

Just me (Sir to you)
Thursday, June 19, 2003

Yes. Managers succeed in estimating development time correctly in only 26 percent of cases.

.
Thursday, June 19, 2003

Inside Outsourcing in India
http://www.cio.com/archive/060103/outsourcing.html

At the bottom of the article you can read comments on the article by clicking on a link.


Thursday, June 19, 2003

I work for a multinational firm in South America and I have been involved in several offshore development projects in the last 2 years. Let me present a (perhaps too) candid description of the results:

- all projects had high rework rates, from 25% up to 80% of development time

- all projects were delivered late, taking 10% to 100% longer than expected, mostly because of rework

- all projects were much cheaper than if they were done in the USA. On average, they had a 60% higher margin, although they were sold by half the price. Even more surprising, we were able to break even in the (fixed price) project that was redone 4 times and took twice as long to be finished, although 5 or 6 American managers were assigned to it mostly to save their jobs.

Analista
Thursday, June 19, 2003

Seems resonably consistent with the number given in the presentation above:

Average cost overrun: 186%
% of projects that require a restart: 94%
Average time overrun: 222%
Functionality delivered on average: 61%

Ultimately:
28% of projects fail completely
46% of projects end "challanged"
26% of projects succeed

Let me restate that these are NOT the numbers for outsourced projects, but for large projects in general. I merely give them to illustrate that stating "I know some outsourced projects that failed" might not be indicative of specific problems with outsourcing, but regretfully is more an indication of the state of our industry in general.

Just me (Sir to you)
Thursday, June 19, 2003

Our company has two outsorced projects...

Project 1 is on version 2. The first version was a disaster. Specs were changed 2/3 of the way into development, the team of 15 developers changed numerous times, defects were only closed 40% of the time.

Some of the documentation was written by the developers and some by us. Requirements were often misunderstood, the documentation no longer matches the end result and the database design is all over the place.

The developers cost a third of the price, but cost four times the actual effort and the application works barely.

I have taken over for version 2. The team was slimmed down to 5 people, who could not be moved from the project. They do not write documentation, we do, they just point out all the faults and ask for clarifications. We correct them and add them to a change log. We have a forums for technical questions, so that a knowledge base is built and important information does not get lost in e-mails.

More importantly, we are bang on schedule, with fewer defects and a closure rate of 85%.

So, it's not the outsourcing that is the problem, but the ability to manage and communicate from our end. I've learnt from 3 years of experience.

However...

Project 2 is managed by someone else. The specification is to create a Change Management product that 'works like product x' and bolts into my product. Even this is not documented.

The first step is to create a prototype, but as yet, no-one has requested the data structures for my product. The prototype is already late and I have no doubt will have to be corrected significantly, since it can't really correct other than by pure fluke.

Nobody actually knows how it is going to work and despite finally getting myself on the project board (since I have ALL the key technical knowledge), I've yet to be consulted.

They were supposed to be launched in parallel. Mine is in testing and due, on time, 8 weeks. The spec for the next version is written, so that two people can start on the R&D

The other one is still at the prototype stage and will not have a technical spec for another 4 weeks. My guess is that it will take 3-4 months development, but no-one has any idea on the size or make-up of the team.

By managing the outsourcing correctly I may actually crank out two good quality versions of my product in the time it takes the other project to crank out one bad version.

Pete J
Thursday, June 19, 2003


Great reading on the replies.

Unfortunately I don't see this as a bandwagon trend but as an IT manaagement  trial and error process that eventually they will perfect or at least get to a point where cost savings are substantial and project time over-run is minimual ; think of it like RUP iterations :-)

There are already hybrid IT consulting houses that can offer a combination of onshore and offshore development.  It's viable as there are ways to mitigate the offshore pitfalls:

- keep a small core team onshore to technical manage/architect and roll-out/QA product when ready
- part of onshore architect team is bilingual
- force remote staff to work north american hours
- outsource cost of office space/dev staff to a firm specializing in that in the region (pay $X/dev/month etc.)

Pure development will be shifting offshore I think.  In the long run architecture will as well and perhaps offshore countries will become the primary suppliers of software in the future.

James Nicoll
Thursday, June 19, 2003

A late product can mean a missed opportunity, which is a LOT more costly than any programmer-wages.

Of course, the jurry's still probably out as to if a domestic team is going to be more or less likely to bungle a project than an offshore team.  ;)

Flamebait Sr.
Thursday, June 19, 2003

Actually,

A few "near-shore" develoment outfit are starting to pop-up around here in Montreal.  The cost of a developer is roughly 25%-35% of the cost of a developer in a major US city.

Same language (english is very common), same time zone (at least with the East Coast), similar culture and the ability to fly over quickly makes it a somewhat competitive alternative to offshore development.  The phenomenom seems to be geting some traction lately.

Claude Remillard
Thursday, June 19, 2003

Just me, different criteria are in use when people talk generically of projects that, allegedly, fail, and when they talk of por outsourcing results.

The first usually involves business managements making pronouncements based on ROI, achievement of deadlines and similar objectives without regard to the validity of those criteria. Those judgements are not, in my view, well informed or useful.

The second set generally consists of developers making informed judgements on product quality of outsourced projects. Those judgements are different from the first set, as well as being better informed.

.
Friday, June 20, 2003

.

I am not so sure about both the 1 not being usefull as well as the 2 being better informed.
Things like ROI seem quite usefull to me and I know very few developers that can actualy judge the quality of code produced by other developers. A lot of the time you get a "this is unworkable garbage" reaction to everything except their own/home team codebase.

Now do not get me wrong here. I am not trying to argue that offshore is heaven and that we should all go out and pick something to outsource today. I believe very few organizations have the skill and process nescessary in house to be successfull at this game.

Just me (Sir to you)
Friday, June 20, 2003



One big issue I see with offshore development is the distance/language barriers between the domain/business knowledge folks.  It forces projects back to a full waterfall model where they need bullet proof requirements and architecture prior to engaging the resources.

Either that or you puposely phase your project to include a short prototype phase where you test out what the remote team can build etc.

James Nicoll
Friday, June 20, 2003

A major Canada/US corp sent some developer jobs to india, and fired two well-seasoned American developers.  After 8 months of indian blundering, failure to understand the problem, overwriting code directly into the source repository (!), and failing to fix a simple bug with an HTTP cookie, the company decided that it's outForcing whore, Cheap Skate Corp, should hire back the experts. 

The bug was fixed by the re-hired developer in less than one hour.

Xavier Breath
Wednesday, April 07, 2004

*  Recent Topics

*  Fog Creek Home