Running an oncall rota

Being part of an oncall rota is pretty much a certainty if you work in the systems or operations team at any IT orientated company. Stuff needs to be working 24 hours a day, 7 days a week and therefore you need someone available to apply duct tape and staples when things go wrong out of normal working hours. It continually amazes me, therefore, that so many companies get the management and organisation of their rota badly wrong, sometimes to the point of it being a significant factor in staff moving on.

As someone who’s participated in oncall rotas at all levels, here’s how I think one should be organised:

  • Sysadmins should only participate in the rota after they have completed their probationary period. Think of the goal of the probation period as being getting someone up to speed so they can participate.
  • Each oncall shift should last one week, and rotate at 2.00pm on Tuesday. If you run a late shift, i.e. 11.00am to 8.00pm, make the person on that shift also be oncall. Get all the unsociable hours out of the way in one lump.
  • The bare minimum gap between oncall shifts should be 5 weeks. If your rota is shorter than this, then you don’t have enough qualified staff to cope with holidays, illness, paternity/maternity leave etc anyway.
  • The oncall shift should be mapped at least 3 months into the future. Staff should be free to swap oncall weeks providing that no-one is ever oncall two weeks in a row, and that all swaps are cleared with their line manager.
  • Issue the oncaller with a good quality mobile phone and 3G capable laptop. The phone should not be a smartphone, the aim should be for maximum battery life and talk time. The laptop should be more like a netbook than a desktop replacement, should come with a spare battery, and should dual boot to Windows and a useful Linux desktop distro. These days I’d shell out for a nice big SSD to stick in it too. Make sure there are no restrictions on international calling and no data caps for the phone and 3G card.
  • Don’t give the direct number of the oncall phone. Instead, use an answering service as filter.  During office hours, the answering service should redirect enquiries to the regular helpdesk/support number.  Out of hours, the answering service should accept calls, take details and then pass these on to the oncall number. Additionally this service should text and email call details so there’s a record.
  • Pay a fixed daily amount for being oncall. Pay 1.5 times this amount for being oncall on weekend days. Pay 2 times this amount for being oncall on a public holiday. Where someone is oncall on a public holiday, add one day to their holiday allowance.
  • Pay a per-incident fee when oncall is used. Each oncall use should be tracked in your ticketing system. Make using oncall a business cost, thus giving the business a reason to make sure oncall is not used trivially, and a reason to make sure problems are fixed permanently and not just temporarily alleviated.
  • When the oncall person has dealt with out of hours issues, don’t expect them in at the regular time the next day.  Expect them to use their judgement to make sure they are suitably rested.   You do not want an overly tired oncaller dealing with problems on production systems.
  • The person oncall should never be taking on “out of hours” work. Want a database dumped and reloaded? Want a disk unmounted and fscked overnight when a server’s not busy? All those things can be done, but not by the oncall person. That person is there to respond to problems, not to perform routine or planned maintenance.
  • Make it very clear that abuse of oncall is unacceptable. Oncall is there to fix customer or service affecting problems, not to help someone with Excel.
  • Have a clear demarcation between production and testing/development/QA systems. The latter group are not oncall’s responsibility to fix.
  • If you have offices around the world, have a “follow the sun” oncall system, get your offices to cover each other.
  • Have realistic expectations of oncall response times. If you need to guarantee that problems are attended to within 20-30 minutes then you should be running an overnight shift, not oncall.
  • Expect a daily report summarising the previous 24 hour oncall period, even if that report is “Nothing to report”. The weekend period could be lumped together on the Monday.
  • Have a weekly oncall handover meeting between the outgoing and incoming oncall staff.
  • During the day, have a junior or trainee sysadmin be oncall. It’s good practice for them.

Doing all of the above shows that you take oncall seriously, and you appreciate the impact being oncall has on someone’s life. Your oncall staff are the people who salvage the business’s reputation when the midden hits the windmill at some unworldly hour of the night. Keeping them happy and making them feel valued and respected can only be to the business’s benefit.

Why I won’t be voting BNP on May 6th

THINK; Do you really want to vote for despicable parasites who prey on the innocent people?”

….screams the headline of the HAVERING PATRIOT, the local BNP party’s leaflet stuffed through my door.  “Of which despicable parasites do you speak?”, I ask.  Luckily the leaflet provides examples.  Reading though, and  roughly checking the allegations presented, I would broadly have to agree with them that, no, I am unlikely to want to vote for any of them. Of the 17 people listed:

Three I will not vote for because:

  • They are Conservative candidates.
  • They are not standing in my constituency, I couldn’t vote for them if I wanted to.

Thirteen I will not vote for because:

  • They are Labour candidates.
  • They are not standing in my constituency, I couldn’t vote for them if I wanted to.

The remaining candidate is Frank Beck, (actually, they say Becks), a Liberal Democrat and thus escaping the first sorting, but I still won’t be voting for him for three compelling reasons.

  1. He’s not standing in my constituency, I couldn’t vote for him even if I wanted to.
  2. He’s not standing for election at all, this makes voting for him tricky.
  3. He’s dead.  The final and most fatal nail in the coffin.

I read the leaflets, 3 of them, quite carefully.  (Often painfully, as it seems spell checking and basic grammar were clearly less important than wiping all the rabid foam off). Some of the new words and phrases invented are quite good,  though I think they’ll have trouble with “tri-axis of treason”, quite a few people nouned that before them. The presentation is a confusing mix of random italics, bits in bold for no `real` good reason, odd font sizes, varieties of quotes in which the most mundane axioms are headlined at you. “Vote BNP, The party made of real Men and Women.”  Well, err, yes, I accept you’re probably real, can’t fault you on that one.

They have a picture of some kind of machine pistol on the leaflet.  None of the text in any way refers to this picture.

One of the leaflets states two alledged facts.

“Fact #1: The Conservative Party, the Labour party and the Liberal-Democrats all want Turkey to join the EU.”

On the face of it, this seems broadly true.

“Fact #2: The BNP is the only party which opposes Turkish membership of the EU and the Islamification of Britain.”

Oh really?  (Emphasis mine).  I confess I’m struggling to find a link between Turkey joining the EU and the  likelyhood of Britain being swayed en masse to convert to Islamic culture.  I like bacon, sorry.

Please do vote on May 6th, please don’t vote for the BNP.

Please vote on May 6th

I don’t often make public political statements, and saying “please vote on may 6th” probably isn’t one really.   However, it is vitally important that you, the enfranchised public, get off your apathetic arses and make it to the polling station this coming May 6th.

According to this information almost 4 people in 10 eligible to vote chose not to bother to do so. Oh there will be a few spoiled ballots in there but most of that 40% will simply have not bothered.  That’s shocking.

I do feel Australia has this nearly right.  It’s compulsory to vote there.  You can turn up and spoil your ballot, I understand drawing pictures of penises on the form is common, but the point is that you do show up, or you get a fine.  It’s not much, personally I’d be quite happy with the penalty for not voting without good reason is having your right to vote removed thus making my vote worth that tiny bit more, but it’s better than what we have now.

Why turn up just to spoil your ballot?  Well if the result of the election show that 40% voted “none of the above” then that’s a clear message to the politicians that their policies and performance are not appealing to a significant proportion of voters. Maybe it’ll make them wonder why we’re not interested in supporting them.

Another reason you should vote is not to get a candidate in, but to keep a candidate out.  The BNP got power not so much because people voted them in, but because people didn’t vote against them.  We do not need the BNP in this country. Where you have a BNP candidate standing, and you have no preference, voting for anyone else is better than you not voting at all.

Me?  I’ll be voting Liberal Democrat this year, as I have done in every election I have been able to take part in.  I don’t think Nick Clegg is the best LibDem leader ever, I also do think him doing well in the leadership debate was nearly inevitable.  Only a complete cock-up could have done anything but boost the LibDem’s cause.  If I felt either a Labour or Conservative government were really able to sort things out, I’d be voting them in.  I have lived my entire life under either a Conservative or Labour government, things are not going all that well.  No-one voted for our current Prime Minister. Can the LibDems really do any worse?

You may not think it matters who you vote for, I would disagree. But really, really, it does matter that you do vote.

Please vote on May 6th.