Tag: sysadmin

When contract hunting goes wrong: TEKsystems & Allegis Group

I was approached by a recruiter from TEKsystems who were looking for a Linux systems administration and automation type person for a project with one of their clients.  I took a look at the job description, and it seemed like a pretty good match for my skills, so I was happy to apply and for TEKsystems to represent me.

I was interviewed three times by members of the team I would be working in over the course of about two weeks.  The people were based in Sweden and Norway and, having previously lived in Norway, I felt brave enough to try out bits of my very very rusty Norwegian.  The interviews all seemed to go well and, a few days later, I was offered the role which I accepted.  A start date of May 15th 2023 was agreed.

I consider it a sincere and meaningful compliment when I am offered work, so it’s important to know that, in accepting this role, I had turned down three other opportunities, two permanent roles and one other contract.

As this role was deemed inside IR35, I would have to work through an umbrella company.  It’s usually less friction to just go with the agency’s recommended option which was to use their parent company, Allegis Group.  I duly went through their onboarding process, proving my address, identity, right to work and so on and so forth.  All pretty standard stuff.

As May 15th approached, I was conscious that I had not, as yet, received any initial onboarding instructions neither directly from the client or via the agency. Whom did I contact on the 15th, when and how?  As this was a remote work contract, I was also expecting delivery of a corporate laptop.  This had not yet turned up.

Late in the week before the 15th, I had a call from the agency saying that there had been some kind of incident that the team I would be working with had to deal with.  They had no-one available to do any kind of onboarding with me, so would I mind deferring the start of the contract by a week?

It turned out it was very convenient for me.  A friend of the family had died a few weeks earlier from breast cancer and the funeral was on the Friday beforehand and, as it happened, my wife and daughter also got stranded in France due to the strikes.  A couple of extra days free to deal with all of that were helpful, so I agreed and everyone was happy.

Towards the end of that week, there had still been radio silence from the client. The agency was trying to obtain a Scope Of Work from them which would lead to an actual contract being drawn up for signing.

The next Monday was a bank holiday and, on the Tuesday morning, I got this message from the agency.

Hello Martin

We would like to update you to confirm we are unable to continue with your onboarding journey, and as such your onboarding journey has now ceased.

We wish you all the best for your future assignments.

Many thanks,

OnboardingTeam@TEKsystems

Needless to say, this was rather surprising and resulted in me attempting to get in touch with someone there to discover what was going on.  No immediate answer was forthcoming other than vague mentions of difficulty with a Swedish business entity not being able to take on a UK-based resource.  I was told that efforts would be made to clarify the situation.  To the day of writing this, that’s still not happened.  Well, not for me at least.

At the end of that week, it became obvious that whatever problem had happened was terminal for my contract, so I started back contact hunting and reactivating my CV on the various job boards.

I asked TEKsystems if they would offer any kind of compensation.  I’d acted entirely in good faith: I’d turned down three other offers of work, told other agencies I was no longer available and deactivated my CV on the various job boards.  It seemed fair they should offer me some kind of compensation for the lost earnings, wasted time and lost opportunities.  They have declined this request leaving me entirely out of pocket for the 3 weeks I should have been working for them and, of course, unexpectedly out of work.

I’m obviously back looking for my next opportunity and I’m sure something will be along in due course.  This is a cautionary tale of what can go wrong in the world of contracting and, if your next contract involves TEKsystems or Allegis Group, you might wish to be extra careful, making sure they are actually able to offer you the work they say they are, and that you get paid.

Buying a custom gaming PC from Overclockers UK

My current workstation and gaming PC is slowly disintegrating. I built it myself from components some 6 years ago and it’s simply wearing out.  Several USB ports don’t work and Windows sometimes bluescreens with errors that suggests bits of the motherboard are getting tired.  I don’t really have the spare time needed to build a high-end PC and make a great job of it, so I decided to treat myself to a pre-built custom system.   After hunting round, I settled on Overclockers as the company to buy from.

Their system configurator didn’t quite give me what I wanted, so I contacted them and asked if they could customise further which they could.   I put together my list of requirements, they send back a price.  I paid, cash wired to their bank account, upfront and sat back and waited for my new shiny liquid-cooled PC to arrive.

The system shipped.  It shipped to the wrong address.  I had provided Overclockers with a billing address and a shipping address.  They shipped to the billing address which is almost guaranteed to be unoccupied during regular working hours.

A simple mistake.  It happens.  I contacted the courier who were unable to redeliver again that day, but promised they would deliver it to the shipping address the next day.

Next day, my new shiny PC arrives.  I opened the smaller of the two boxes, one for spare components and so on, and immediately see a problem.  The spares and cables and whatnots are not branded with anything I specified, wrong motherboard and wrong graphics card.  I call Overclockers who suggest that the component boxes may have been mixed up and can I please open the main box and check. I do.  It’s someone else’s computer.  I later learn that my system has been shipped to somewhere else.  Overclockers’ mistake?  Courier’s mistake?  It doesn’t really matter. Overclockers have a courier come and pick up this system.

Meanwhile, my system makes its merry way back to Overclockers’ HQ and I, confusingly, get an email asking what I’d like done with it.   I suggest shipping it to the shipping address and could I please have an AM delivery so I don’t potentially waste a whole day.  I offered to pay for whatever that was going to cost.  Overclockers said it was no problem.  Super.

My PC finally showed up at Friday 8pm.   The more astute amongst you will spot that 8pm is not exactly an AM delivery.  Overclockers’ mistake?  Courier’s mistake?  I have no idea, the question has not yet been answered.

I unpack my new PC.  The first thing I notice is that there is a bolt rolling around in the bottom of it.  Stuff can come loose in shipping, so what.  I find that the bolt belongs to a radiator housing in the bottom of the case, there’s a hole, a loose radiator and tool marks around the hole.  Not ideal, but the system’s not going to be moved around much so no big problem.  Despite being an SLI system, there was no SLI cable installed linking the graphics cards.  Simple to fix, but a silly thing for an expert system builder to miss.

One of the customised things I asked for was the pre-cabling of some SATA drives bays: one for a blu-ray writer and two for a pair of big SATA disks I use for bulk local storage.   None of these were done.   I call Overclockers about this, and the loose bolt, and they say there’s not much that can be done without returning the system to them.  As I’ve no interest in another game of couriers, I grumble a bit but then do the cabling myself.

Over the next day or so I had almost no chance to really push the new system.  It ticked over happily, was lovely and quiet and lovely to look at too.  On Sunday night, though, the headphones went on, the office door was closed and I got on with a bit of GRID 2, with all the visual effects turned up to maximum.  I settled down for a couple of hours of hard racing.   After about an hour, the screen froze, went black, and all the system fans kicked into life.

I powered off, reached for my mini-torch and opened the case.  What I saw sickened me: liquid coolant leaking from the CPU block, down onto a graphics card and spilling on to the motherboard.  It was impossible to tell whether the CPU had simply thermally shut down or if the coolant had shorted something expensive.  It kind of didn’t matter.

The next morning I called Overclockers who arranged to pick the system up.  I asked if they could sort of the cabling and the loose bolts while they were at it.  They agreed.

A couple of days later, I got an email saying the system had been repaired and was on its way back to me.  The next evening I get a call from the owner of the billing address saying that a courier had tried to deliver something with my name on it.   They had shipped to the wrong address. Again.

I had now run out of patience and I asked for a full refund.   To their credit, Overclockers didn’t argue on this and they said one would be arranged.  As it was convenient for me, I asked to keep the Windows 8.1 licence and the SSD.   As it was convenient for them, I agreed to pay for these again separately, they would then issue a refund for the full amount of the original transaction.  I didn’t ask, but I kind of expected they would simply wire the cash back to my bank account.

After 3 days or so, nothing had showed up, so I called and they said that processing a refund might take up to 7 working days.

Today, 10 days on, nothing had showed up, so I called them and they said a cheque had been issued on the 4th and had been sent to……. you guessed it, the wrong address.  The owner of the address had not had a cheque arrive.

They offered to send a new cheque to the right address.  I suggested they simply wire the money to my account, I was told this was impossible due to the people who would have to do that being in Germany.  No, makes no sense to me either.  I asked if the cheque could be sent by special delivery, for which I was happy to cover the costs.   This was, of course, not possible.

So, 5 weeks after placing a cash order for a high-spec custom PC from Overclockers UK, I have no PC and they have a large amount of my money.

Please consider this post next time you’re thinking of ordering from them.

 

(Update: 15/4/2014:  A handwritten cheque arrived from Overclockers this morning. )

Communicado update: A change of tactic

The work to make Communicado’s life as difficult as possible continues and it does seem like we’re having some success.

When I started this project, Communicado registered all their domains through DAILY mostly using faked registrant data and hiding behind the privileges granted to individual private registrants.  I established a dialog with Nominet about this and it seems Nominet did take action to the point of suspending some of these domains.   Communicado then suddenly switched to using ENOM for registering their domains, I don’t know and have no way of knowing if they were booted off by DAILY or just decided to switch.  Either way, it made no difference, I could easily find the domains they were registering via Nominet’s PRSS tool.

As of Monday 16th, they have changed tactics again.  They have apparently abandoned the .co.uk namespace (I’m sure they’ll be missed) and have gone back to using a variety of .com, .net and .org domains.  Some seen in use today are:

actionallegiance.com
andronol.com
baotao.org
bigrockconsultants.com
coolpress.net
europacastno.com
greenroses.org
hourlycreative.com
pidchas.com

They’re easy enough to spot in the logs, but I don’t currently have a good way of searching the whois for these TLDs. Suggestions for such a tool (non-free is fine) are welcome.

Maintaining this list and the RBL service is taking time and money.  I will absolutely never be charging anyone for the list and the RBL will be free and open access for as long as it is sustainable to do so.  In addition to the ways you can help mentioned in previous posts, a more direct way you can help is to donate a little money, preferably in the form of Bitcoin to 1F9Y1Gd3Pmmchxa7uGFd3zBQY9zVuX78Jd.

More news when I have it, you can follow @Excommunicado for more frequent updates.

An update on Communicado

It has been a busy few weeks since I first blogged about Communicado, here are some of the highlights of what has been going on.

  • Communicado are still registering somewhere between 40 and 60 new domains a week.  The blacklist is being regularly updated and currently has 5364 domains listed.
  • Communicado appear to have switched registrars from DAILY to ENOM as of yesterday.  Makes no difference to picking up their domains.
  • Nominet has been investigating and tell me that some of Communicado’s domains have been suspended and they are in the process of suspending more.
  • Please follow @Excommunicado for news and announcements on Twitter.  Low volume, only on topic.
  • The existing text file download will continue to be updated but, by popular demand, I have set up a DNS RBL containing their domains.  As of the time of writing it is open access, that may change if it becomes too busy.  Using it is easy:
martin@olga:~$ host malimanosa.co.uk.excommunicado.co.uk
malimanosa.co.uk.excommunicado.co.uk has address 127.0.0.2
martin@olga:~$ host flobbletob.co.uk.excommunicado.co.uk 
Host flobbletob.excommunicado.co.uk not found: 3(NXDOMAIN)

If anyone wants to provide working configuration examples for SpamAssassin (or other similar tools), I will cheerfully link to them or post them here.

More news when I have it, have a Communicado-free afternoon!

Cooperative Energy and password security

As a protest vote against the Big 6 energy companies, I recently switched supplier to Cooperative Energy.   Switching is painless, fill your details in online, click the button and off you go.   They do of course want a password from you and I used LastPass to generate a unique one for me and memorise it.

Some time later, I went to login in to the customer portal just to see what I could do and was quite surprised to find my password didn’t work.  I mentally shrugged and clicked on the Forgotten Password link and waited for the usual password reset email to arrive.  I got this instead:

Dear Customer

The information you requested is…
eg!3fpp*hvfs

If you have any questions please contact our customer service team

(This is, of course, not my actual password, this is just an example that I’ll treat the same way as the Coop did.)

Here we have two immediate problems.  The first is, of course, they have sent me my password in plain text in an email.  We all know that’s a bad idea.  Secondly, what they have sent is not actually my password.  My password looks like this:

eg!3fP^P*hVFs

See what they did?  For whatever reason the caret has been removed and all the letters have be converted to lower case thus making my password less secure.    I sighed and went to change my password online and found I couldn’t.   If I want to change my password then I have to go talk to a human to do so.   This leads to problem three, which is that people generally pick stupid passwords and reuse them.   I’m sure Coop Energy only employ wonderful honest people, but giving them an email address and a stupid password is only ever going to end badly for someone eventually.

I’ve spoken to Coop Energy’s customer service team and they acknowledge the problems I’ve found.   Let’s hope, for the sake of a safer and more secure internet, they sort them out.

Running an oncall rota

Being part of an oncall rota is pretty much a certainty if you work in the systems or operations team at any IT orientated company. Stuff needs to be working 24 hours a day, 7 days a week and therefore you need someone available to apply duct tape and staples when things go wrong out of normal working hours. It continually amazes me, therefore, that so many companies get the management and organisation of their rota badly wrong, sometimes to the point of it being a significant factor in staff moving on.

As someone who’s participated in oncall rotas at all levels, here’s how I think one should be organised:

  • Sysadmins should only participate in the rota after they have completed their probationary period. Think of the goal of the probation period as being getting someone up to speed so they can participate.
  • Each oncall shift should last one week, and rotate at 2.00pm on Tuesday. If you run a late shift, i.e. 11.00am to 8.00pm, make the person on that shift also be oncall. Get all the unsociable hours out of the way in one lump.
  • The bare minimum gap between oncall shifts should be 5 weeks. If your rota is shorter than this, then you don’t have enough qualified staff to cope with holidays, illness, paternity/maternity leave etc anyway.
  • The oncall shift should be mapped at least 3 months into the future. Staff should be free to swap oncall weeks providing that no-one is ever oncall two weeks in a row, and that all swaps are cleared with their line manager.
  • Issue the oncaller with a good quality mobile phone and 3G capable laptop. The phone should not be a smartphone, the aim should be for maximum battery life and talk time. The laptop should be more like a netbook than a desktop replacement, should come with a spare battery, and should dual boot to Windows and a useful Linux desktop distro. These days I’d shell out for a nice big SSD to stick in it too. Make sure there are no restrictions on international calling and no data caps for the phone and 3G card.
  • Don’t give the direct number of the oncall phone. Instead, use an answering service as filter.  During office hours, the answering service should redirect enquiries to the regular helpdesk/support number.  Out of hours, the answering service should accept calls, take details and then pass these on to the oncall number. Additionally this service should text and email call details so there’s a record.
  • Pay a fixed daily amount for being oncall. Pay 1.5 times this amount for being oncall on weekend days. Pay 2 times this amount for being oncall on a public holiday. Where someone is oncall on a public holiday, add one day to their holiday allowance.
  • Pay a per-incident fee when oncall is used. Each oncall use should be tracked in your ticketing system. Make using oncall a business cost, thus giving the business a reason to make sure oncall is not used trivially, and a reason to make sure problems are fixed permanently and not just temporarily alleviated.
  • When the oncall person has dealt with out of hours issues, don’t expect them in at the regular time the next day.  Expect them to use their judgement to make sure they are suitably rested.   You do not want an overly tired oncaller dealing with problems on production systems.
  • The person oncall should never be taking on “out of hours” work. Want a database dumped and reloaded? Want a disk unmounted and fscked overnight when a server’s not busy? All those things can be done, but not by the oncall person. That person is there to respond to problems, not to perform routine or planned maintenance.
  • Make it very clear that abuse of oncall is unacceptable. Oncall is there to fix customer or service affecting problems, not to help someone with Excel.
  • Have a clear demarcation between production and testing/development/QA systems. The latter group are not oncall’s responsibility to fix.
  • If you have offices around the world, have a “follow the sun” oncall system, get your offices to cover each other.
  • Have realistic expectations of oncall response times. If you need to guarantee that problems are attended to within 20-30 minutes then you should be running an overnight shift, not oncall.
  • Expect a daily report summarising the previous 24 hour oncall period, even if that report is “Nothing to report”. The weekend period could be lumped together on the Monday.
  • Have a weekly oncall handover meeting between the outgoing and incoming oncall staff.
  • During the day, have a junior or trainee sysadmin be oncall. It’s good practice for them.

Doing all of the above shows that you take oncall seriously, and you appreciate the impact being oncall has on someone’s life. Your oncall staff are the people who salvage the business’s reputation when the midden hits the windmill at some unworldly hour of the night. Keeping them happy and making them feel valued and respected can only be to the business’s benefit.