Fog Creek Software
Discussion Board

Last Straw

I am in charge of producing some graphs for a report that my company is legally required to present to a government agency, every quarter, on pain of prosecution.

I am trying to be concise but my blood is boiling and I'm also tired, so I'm not succeeding. Sorry. Anyway:

The data that goes into the graphs is very simple: for every deal over the last 12 months, the count of each item sold and the percentage discount. This requires only eight pieces of data, all of which could be expected to be readily available in any company under normal circumstances. Unfortunately the information systems in my company are far more incoherent than you'd expect, but that was only part of the problem.

From the moment I inherited this piece of work I have encountered shocking behaviour from most of the stakeholders, ranging from incompetence to passive resistance to lies, and even - I suspect - deliberate sabotage. The programmer before me handed over a manual process which he assured me was accurate, but which proved to be anything but. The instructions were incomplete and largely erroneous. None of the scatterware he left me worked at all - I don't mean it was buggy, I mean it flat-out didn't run. I ended up spending three twenty-hour days getting the process to work in time for the deadline.

Why did I leave it so late, you may ask? The answer is, I didn't. A major part of the process was the job of business people in the various departments, but ExProgrammer had been doing many hours of unpaid overtime to (manually) do that job for them. Before his departure, he did leave (broken) Access apps so that the relevant staff could enter the four pieces of data they needed (once I fixed them). The problem was, they didn't want to do it and ran the gamut of passive-resistance antics from ignoring my requests, to lying, to demanding (literally) that I train them in how to select an item from a drop-down list (really, no kidding) to accusing me (a week before the deadline) of never having mentioned it to them at all (I had an inch-thick wodge of correspondence that proved otherwise) to saying (four days before the deadline) "why can't we have it done the way ExProgrammer used to do it?" Even if I had been allowed and willing to do as he did, I couldn't have - he was the only one who knew how to do it and he didn't document any of it.

Over many months, and with great difficulty, I automated the process until it could be done at the touch of a button in under three minutes. Life was great for two whole weeks. Then:

The data source I previously used is being replaced with a massive CRM initiative. But since I only need eight pieces of data this should be no problem, right? It would be BigCorp's job to write extract scripts according to my requirements.

Months of design meetings. Months of documentation amendment. Months of rewriting my graph-producing app. More changes. More meetings. More rewriting. Having to accommodate bad design such as two fields being concatenated into one so I would have to split them, and then another two fields being concatenated, but without telling me, so I wasted time figuring out why the accuracy was fucked up and rewriting my graph-producing app yet again.

Oh, did I mention that BigCorp don't know SQL well enough to do joins? So the scripts take 3 hours because they loop through each row and compare it with each of 45,000 other rows to see if they match. But they do work accurately, if slowly, and anyway I'm not allowed to change them because BigCorp own them now.

Finally, tested and signed off. Days later it was mentioned that one of the eight fields was being radically changed so my scripts wouldn't work any more, but not to worry, BigCorp would take care of it.

I got suspicious and asked about it. BigCorp claim that they made the change specifically to benefit me, though of course they didn't consult me or the business owner, the only two people who fully understand the process. They just decided that my requirements had radically changed and, with uncharacteristic despatch, implemented that change. They do not grasp that the extract scripts will now not work unless they rewrite them (I'm not allowed to, remember?) because, in fact, my requirements *haven't* changed. At all. I have no idea why they would think any such thing.

Besides, we've already fought to go through weeks of change control because of the new status codes which are crucial to the accuracy and which I also heard about by chance, through rumour. Five weeks to the next publication deadline; it's touch and go whether they'll be able to change that single line of code in time. Surely it would be outrageous to ask them to make ANOTHER change when the fate of the CRM project as a whole hangs in the balance, just because our managing director will get prosecuted if the graphs are wrong and probably try to take me down with him.

Oh, and did I mention that each and every one of these eight fields has been changed in a similar way? Oh, and there's wrong data in one of the fields which will also screw the accuracy to blazes? If they're in this situation two weeks before the final deadline, after weeks of release, after well over two years on the project, why should this chaos not continue until the end of time?

It may be true that BigCorp has the responsibility for making their scripts work and will carry the can if the graphs don't come out right. It's just that I'm not sure that will actually achieve much at 11:59 pm on the night before the deadline, when I'm sitting there staring at an obviously inaccurate graph and a pile of insanely written extract scripts that I'm not legally allowed to change.

By no stretch of the imagination can I afford to resign, but I'm strongly tempted to do so if that's what it takes to get rid of this albatross. My colleagues continually reassure me about what I can get away with, but that's not all it's about. I just don't see how I can work in this funny farm for another minute and stay sane. In comparison, bankruptcy seems appealing.

Fernanda Stickpot
Friday, July 16, 2004

Sadly, this doesn't sound much different from any large corporation I've worked in. Being able to cope with idiocy is one of the most important skills a programmer can learn. In a large bureaucracy it may be the most important skill.

Saturday, July 17, 2004

I expect you're right. It's just that after 18 months of this I'm knocking back the Thorazine like M&Ms.

On Monday I'll write ANOTHER urgent memo stating:
- this is where I understand the process to be
- this is what needs to change so that the process can meet the requirements
- this is the deadline for delivering the changes
- this is not negotiable

And that, now that they know this, I trust that the responsible parties can be relied upon to take appropriate action, and, if any of the relevant data changes in the future, to inform me immediately and consult me as to whether this will be an obstacle to the company's meeting its legal obligations, rather than allowing me to find out through the unacceptable medium of 'the grapevine'.

And that any dispute over the requirements can be taken up with the business owner, who will in turn discuss them with me.

And with that, I shall smartly adjust my hat, pull on my white gloves, and depart to take the waters in Ischia before returning, refreshed and restored, to the bowels of hell.

Fernanda Stickpot
Saturday, July 17, 2004

You need to get into the habit of either not promising specific deadlines, or telling your superiors that any changes to the process will change that deadline - and be firm about it.

Matthew Lock
Saturday, July 17, 2004

Matthew, I don't think he has any control over the deadline whatsoever.  I believe it is an externally imposed hard deadline.

Aaron F Stanton
Saturday, July 17, 2004

What a whining and sniveling brat you are. Have you no sympathy for the multitudes of people who go hungry every day??? Yet you have a boundless bounty still complain about mere triffles...stupid Americans!!!!

anon-y-mous cow-ard
Saturday, July 17, 2004

Fernanda, if the reporting is a government requirement, then some nominated official of the company will be formally responsible for it being sent in.

Contact the government agency and find out who that responsible person in your company is, then send him or her a formal notice that the company will not be able to comply with its reporting responsibilities.

Because sure as hell that person, if ever called to account for late reporting, will blame the incompetent programmer.

Saturday, July 17, 2004

She's said who is responsible; it's her manager. That's what she's worried about.

Stephen Jones
Saturday, July 17, 2004

Sometimes there are no answers, there's nothing an individual could have, or should have done differently and still they'll end up getting shafted.  It occurs to me that there's an increasing likelihood of this being true if the individual is concientious, cares about the work and even maintains a sense of proportion about it.

Its all very well to comfort yourself in the knowledge that one day 'they'll' get what's coming to them (cf Martha Stewart, Enron et al), but that doesn't put bread on the table.

Its undoubtedly true that we could do without half the population of the world, if we could only identify the right half.

Simon Lucy
Saturday, July 17, 2004


No, really I feel for you.  This tops any story of mine.

I think some of the other advice is OK, trying to find the person that really wants this stuff and who actually has some traction, etc.  But my guess is, saving your sanity will come down to one of two things:

1) Find another job.
2) Give up trying.  Say to yourself each morning "I just don't give a shit anymore", go with the flow regardless of what it is, and walk out the door at 5 no matter what anyone says or does. 

Good luck, and I hope you find a better job.

can't advocate this as myself.
Saturday, July 17, 2004

Er, somehow I glossed over the crucially important word "prosecution" in my first reply.  That changes the picture a bit.

Blowing it off probably isn't an option then.  I agree with the guy who said start sending up smoke signals.  Don't hand in any data you know is inaccurate.  Make them key all the old data into the Access app if you have to.

Find a new job.

can't advocate this as myself.
Saturday, July 17, 2004

My manager threw his weight around on two projects this last week, changing (unwritten) specs at the last minute.

My project got done in a frantic day, the other will take the team weeks.

You tell me what he thought he was doing.

dot for this one
Saturday, July 17, 2004

Who is on your side?

The government? (Some government agencies have sensible people. They may be happy to help you/back you up. Alteratively, some are just waiting for something to pounce on.)

The legal department in your company? (If this report doesn't get filed, suddenly they'll have lots of extra work to do defending the company.)

The CEO (new regulations make CEOs reponsible for certain reports, don't they)?

Someone has to be your ally, and hopefully it's not too late to enlist them.

Saturday, July 17, 2004

Oh, I do have one or two people on my side. The business owner is as responsible for the report in her way as I am in mine, though I notice she follows the company trend of never, ever physically signing anything. I get around that by putting a disclaimer at the front of every document, that if no feedback has been received within 10 working days I will assume that the document has been accepted in its entirety. Needless to say, I circulate a new version of the spec whenever it changes, which is about three times an hour. Just as needless to say, it is most unlikely that any of the recipients have ever read it.

My manager is also on my side, but he has no power whatsoever, and is never here, so I'm not sure how that helps. I do have e-mails dating back to the late fourteenth century, which I will print out on Monday.

In the last meeting, I raised the question of this latest change that had been made supposedly for the benefit of this report, but which in fact would completely screw it up. This led to my trying to explain what the requirements were. Since the meeting was about the project as a whole and not solely about this particular report, I didn't have the spec in front of me nor had I memorized every detail.

Suddenly, all the people who couldn't have been less interested before, began little eddies of conversation about what they thought the requirements should be. The general feeling was that the report needs to be changed again, in a variety of new ways. They certainly didn't believe anything I told them about what I knew the requirements to be.

One helpful person (innocent and unknowing, for a change, but how alarming to see the impression she is getting) earnestly explained to me that I needed to get a list of requirements from the business and get them signed off. DUH! As if I'm not busy enough learning to wave bye-bye and tie my shoes.

Fernanda Stickpot
Saturday, July 17, 2004

Wow, this sounds like a previous place I worked at, but that was in the electronic prescription industry.

This sounds like Sarbanes-Oxley compliance stuff. SSDD.

Hire me! Hire me!

Monday, July 19, 2004

Exactly - and if it's SOX there's a great quote I carry around with me for when I deal with this kind of thing.  It's from an old EWeek that was actually useful enough to cut out and keep the paper around - "Quote of the week:  The message is clear: we need to do everything necessary to keep our CFO and CEO out of jail. - Gary Bronson, IT director of Washington Group International on Sarbanes-Oxley."  After everone has a chuckle over the clipping, I then point out to them that they are in EXACTLY that same spot - and would they like to go to their own CFO/CEO and tell them why it shouldn't be done?

Unfocused Focused
Thursday, July 22, 2004

*  Recent Topics

*  Fog Creek Home