Fog Creek Software
Discussion Board




Pager duty for developers?

I started working as a developer at a large financial institution with a lot of in-house software 6 weeks ago. So far so good. Then, a couple days ago, my boss announced that our three-person team will start rotating pager duty. We're only responsible for supporting our team's software. My initial feeling is that pager duty is very much a non-standard duty for a developer and that this should have been mentioned during the interview. I know I could have asked during the interview (and from now on I will!) but I'd never heard of developers carrying a pager. Also, most of our users are 6 timezones ahead of us, so support calls will come in the middle of the night.

So I guess I want to know other people's opinion on this. Am I right in thinking that pager duty is unusual enough for developers that my boss should have mentioned this during the interview? Is pager duty standard for developers in the financial industry? Advice? Warnings?

Thanks.

Someone who prefers to remain anonymous
Friday, July 19, 2002

You make it, you fix it. Or would you rather have a clueless administrator come in and "duuh....it's broken, why don't I hit the reset button." "Duuh...it still doesn't work..my mouse works but I can't get to the website..let me try the reset button again."

Pager duty also goes a LONG way in forcing developers to write quality code. This is especially true if you are working in the financial market where "testers" are considered "a waste of time". Remember, most financial market people are sales people. They are driven by deadlines, demos, meetings,etc. This means constantly changing system which means no time to test which mean you better be a damn good defensive programmers.

Hint for ya, buy a good logging tool and log a lot. When you make updates to the production server, monitor the log for a few hours after your deployment to make sure you didn't do something stupid. This will save you from the early 6am calls from people on the east coast.

Humbug
Friday, July 19, 2002

Not really ...
If it's critical and you created it... then unfortunately, you have to be avialable to support it.... especially if is complex/nontrival.

XP Man
Friday, July 19, 2002

It's not unusual.

A while ago one well-known company licensed our code to include in their OS. We had to wear pagers 24 hours a day, because they recompile the latest sources of the system every night and in case of build errors have to contact 3rd-party developers immediately. Fortunately, they never really called us, AFAIR.

I know some other big companies that have the same standard policy - 24x7 3-rd party developer support. It's quite normal.

Besides, it's quite possible that your boss didn't know about this pager duty when you were interviewed...

Igor Krivokon
Friday, July 19, 2002

They were deceitful not to tell you the job involved 24 hour support. In large organisations, and especially the financial industry, there are usually extra payments for call-outs.

Hugh Wells
Friday, July 19, 2002

I think so too. You can do it, but negotiate. :)

A
Friday, July 19, 2002

Support is necessary, and standard in the telco industry (or anywhere else that promises 24-hour reliability, excluding embedded systems) - someone must do it.

If I take a new employer, one of my questions will be about what hours I will be expected to work (as well as where).

My boss takes most of the pager support on himself. Why should we refuse to do it occasionally?

We use cell phones, not pagers.

Christopher Wells
Friday, July 19, 2002

Ha!  Been there, done that.  Its standard duty for 24x7 systems like the one your on.  Quite a lot of fun to be called in the middle of the night to find an obscure bug in some 10 year old code, isn't it?  Its easy for some of these people to say "you wrote it, you fix it".  That's easy when your still writing your undergrad programs and spouting off on forums.  What they fail to realize is that you probably didn't go in and write a financial system from scratch.  When I had to carry a pager, it was never my own code that broke.  My code was a tiny fraction of a percent of the total code in the system, so invariably it was someone else's code AND THEY WERE LONG GONE. 
I don't think you should jump ship.  Try to make the situation better first.  Every chance you get you should try to make the code more solid and the error messages easier to understand.  Have the programs try to tell the operators what may be wrong. 

the cluetrain
Friday, July 19, 2002

Would I be right to assume you would not be on direct pager duty for the end-users, but on an escalation from the network/sys-admins?

Just me (Sir to you)
Friday, July 19, 2002

Is pager duty standard for developers in the financial industry? Advice? Warnings


Yes, it is.  Pagers are common for developers in Finance.  In many financial firms, developers support what they have written.  Often, many programmers do little new development, and mostly production support of their previously released apps. 

Bella
Friday, July 19, 2002

Back in the old days... ah, before pagers, it was (and in most IT departments probably still is) called... "Callout".

It's the best thing in the world... for bringing developers up to speed on an application system.

"You broke it, you fix it" is one way to look at it.  However, that implies "finding who to blame" which is not a real productive activity in the middle of the night when you are trying to hit an availability or service level agreement window.  I prefer "It's your job, you fix it". 

If you consider your cushy tushy too high of a level to care about the impacts to your user/customer... then I would suggest moving on to your next job where the stress of doing work is more to your liking.

But don't whine about "your interview" and get a clue.

Business programming is a constant battle with change and there was never any promise given to you that things would always stay as they were when you were given the priviledge of walking through the door as an employee.

Your employer will probably be just as disappointed in not noticing the lack of adaptability during your interview.

Joe AA.
Friday, July 19, 2002

Carrying a pager seems to be common to many programming jobs.  The better companies will compensate for being on pager service, although the compensation is often minimal -- maybe $75 a week.  But that is better than nothing.

Comp time for nights you get pages should be a must, with a 4 hour minimum.  Sometimes it takes longer than that to get back to sleep.  And the night calls also wake up family members.  Someone should pay besides me for the hell given when my wife is waken up by the phone at 3 A.M.

Be careful of companies that will use the pager service for thier own profit.  The last company I worked for had service contracts with the customers, and made a bundle when a customer called after hours for support.  The pager carriers were told that our support of the night calls were just part of our jobs, and we would not get a cent.  Soon no one could be reached on the pagers or even the home phones!

A. Coward
Friday, July 19, 2002

Compensation for callout support needs to be handled carefully.  Certainly if the callout is for customer support and the company is making money, there should be some compensation to the individual(s) being called out.

What you don't want it to become... is a "bonus program" that rewards incompetent work.  It is best to focus it on results (callout causes are eliminated) rather than effort expended (but I worked 4 hours on it!!).  AND... I would never pay for callout for the individual directly responsible for the problem.

For most corporate IT departments, the philosophy of callout should be focused on "help me make it through the night" - i.e. get the processing back to running the quickest way possible.  This includes bypassing bad input records and the like. 

Save the application debugging and repair sessions for non-critical times, and fix it once and for all.  Eventually callout will become a non-issue as the application stability improves.

Joe AA.
Friday, July 19, 2002

I'm not paid extra for being called out, but if I work at night I don't in the daytime.

Christopher Wells
Friday, July 19, 2002

If I get a call about work at 3:00 AM, they will talk to the BofH.
http://bofh.ntk.net/Bastard.html

Doug Withau
Friday, July 19, 2002

Joe AA is a troll

regualar poster that saw the light
Friday, July 19, 2002

Just want to make sure.  You can telecommute when paged, right?  No needing to drive in the middle of the night and probably running over a few children?

curious
Friday, July 19, 2002

> You can telecommute when paged, right? No needing to drive in the middle of the night and probably running over a few children? - curious

Supporting an installation 6 timezones away, I'd bet on telecommuting.

Quit meat and drink, live 20 minutes walk from office. Practice being harmless, useful, faithful, polite, happy, hygienic, secure, and free. Live in the shade.

name witheld by request
Saturday, July 20, 2002

It's no problem but in all industries, when you are on call, you are also on-pay (though at a reduced rate of course).

Some companies legitimately need developers on call and pay a fair rate for the service and that's fine. It's a good way to make some extra money.

Of course if there is no compensation for this, look at them and say "Bite Me".

Sarain H.
Saturday, July 20, 2002

"What you don't want it to become... is a "bonus program" that rewards incompetent work."

Joe since you're such a fine programmer that makes no errors and codes in excess of 1000 prefect bug free lines a day, would you come work for us?

We are willing to pay you $8,000/year, which is substantially more than you are making now.

Sarain H.
Saturday, July 20, 2002

And Joe,

Say hello to your husband Angelo for me.

Sarain H.
Saturday, July 20, 2002

I'm a developer and I've always had pager duty.  Here's the compensation for my last two companies.

Current: (Very large company)

0 calls - $100 per week
< 5 hours $250 per week
< 15 hours $500 per week
15 or greater $750 per week.
Only calls handled outside of work hours are counted.

Previous: (1000 employees)
$50 per week + $1 per minute of time spent.

Before that: (15 employees)
No money.  Respond or get fired.  (I left them after 3 months)

Final Note: I know it can be both <5 and <15.  I just don't feel like coding in a forum.

vanguard
Sunday, July 21, 2002

Sarain... do you dispute "you get what you reward"?  Or do you not want to lose the cash cow that funds your incompetence?

I don't get the "husband Angelo" comment.  Must be some fantasy of yours.

Darn it!!  Am I STILL supposed to be pretending that bugs come from some mystical source outside the developer?  I promised to do better in another post... I guess I'll have to keep trying.

Ok... "Bugs are always around everywhere in the magical ether that fills the universe.  The action of a programmer coding disturbs them and traps them in a vast unseen area that exists between the lines of code."

Hmm... could be the start of yet another Mythodology.

Joe AA.
Sunday, July 21, 2002

Be thankful that you have paying customers who depend on your product!

pb
Monday, July 22, 2002

Joe AA,

No one offerered any counterpoint to your argument.  Rather you are simply called a troll?  Your post made sound philosohical sense.    Don't bother figureing it out, joe AA.  There are very intelligent IT people on this thread, but there are many clueless morons as well.  Oh, and i'm sure someone will say Bella is the same person as Joe AA...b/c we have the same opinion on this.

Another point related to yours is not rewarding the NUMBER of bugs caught.  ie:  You are rewarding people who write the shittiest code. 

Bella
Monday, July 22, 2002

Quit. Now. There is absolutely no way what-so-ever you or anyone else should even consider doing this. Ever. Your life is always more important than your job.

Joe AA: It has nothing to do with where the bugs come from, or even customer service. It is to do with what a developers job is - a developer writes code and maintains it _during their working hours_ not 24/7/365.

Mr Jack
Tuesday, July 23, 2002


In my experience companies that require frequent pager duty are also very light on process.  "We don't have time for that crap, we're a nimble business!" 

I've told my manager before why doing something a certain way was a very bad idea.  Waste of breath - he already knew and had had the arguments already.  What counted was that sales & marketing had more pull with the COO.

Therefore, pager duty.

Of course 90% of the developers were idiots.  They thought that the solution to the problem was to demand that we not carry pagers.  Hint for the percentage that are idiots here:  A pager is not the problem, its a symptom.

Solution:  vote with your feet or shut the hell up.

the cluetrain
Tuesday, July 23, 2002


I used to do support for a Bank whose software provider had stopped supporting some server software they used.  Managed to get a deal worth about $2500 a week - which was nice.  You should definately get some compensation for this - are they expecting you to stay sober/stay local/get up at 3am?    You pay the likes of Oracle megabucks for this, they shouldn't expect it for free from you.

Niall
Tuesday, July 23, 2002

Hey vanguard, don't you know it can be both <5 and <15 at the same...oh.

Dang.

Dunno Wair
Tuesday, July 30, 2002

*  Recent Topics

*  Fog Creek Home