Category: sysadmin

When contract hunting goes wrong: TEKsystems & Allegis Group

I was approached by a recruiter from TEKsystems who were looking for a Linux systems administration and automation type person for a project with one of their clients.  I took a look at the job description, and it seemed like a pretty good match for my skills, so I was happy to apply and for TEKsystems to represent me.

I was interviewed three times by members of the team I would be working in over the course of about two weeks.  The people were based in Sweden and Norway and, having previously lived in Norway, I felt brave enough to try out bits of my very very rusty Norwegian.  The interviews all seemed to go well and, a few days later, I was offered the role which I accepted.  A start date of May 15th 2023 was agreed.

I consider it a sincere and meaningful compliment when I am offered work, so it’s important to know that, in accepting this role, I had turned down three other opportunities, two permanent roles and one other contract.

As this role was deemed inside IR35, I would have to work through an umbrella company.  It’s usually less friction to just go with the agency’s recommended option which was to use their parent company, Allegis Group.  I duly went through their onboarding process, proving my address, identity, right to work and so on and so forth.  All pretty standard stuff.

As May 15th approached, I was conscious that I had not, as yet, received any initial onboarding instructions neither directly from the client or via the agency. Whom did I contact on the 15th, when and how?  As this was a remote work contract, I was also expecting delivery of a corporate laptop.  This had not yet turned up.

Late in the week before the 15th, I had a call from the agency saying that there had been some kind of incident that the team I would be working with had to deal with.  They had no-one available to do any kind of onboarding with me, so would I mind deferring the start of the contract by a week?

It turned out it was very convenient for me.  A friend of the family had died a few weeks earlier from breast cancer and the funeral was on the Friday beforehand and, as it happened, my wife and daughter also got stranded in France due to the strikes.  A couple of extra days free to deal with all of that were helpful, so I agreed and everyone was happy.

Towards the end of that week, there had still been radio silence from the client. The agency was trying to obtain a Scope Of Work from them which would lead to an actual contract being drawn up for signing.

The next Monday was a bank holiday and, on the Tuesday morning, I got this message from the agency.

Hello Martin

We would like to update you to confirm we are unable to continue with your onboarding journey, and as such your onboarding journey has now ceased.

We wish you all the best for your future assignments.

Many thanks,

OnboardingTeam@TEKsystems

Needless to say, this was rather surprising and resulted in me attempting to get in touch with someone there to discover what was going on.  No immediate answer was forthcoming other than vague mentions of difficulty with a Swedish business entity not being able to take on a UK-based resource.  I was told that efforts would be made to clarify the situation.  To the day of writing this, that’s still not happened.  Well, not for me at least.

At the end of that week, it became obvious that whatever problem had happened was terminal for my contract, so I started back contact hunting and reactivating my CV on the various job boards.

I asked TEKsystems if they would offer any kind of compensation.  I’d acted entirely in good faith: I’d turned down three other offers of work, told other agencies I was no longer available and deactivated my CV on the various job boards.  It seemed fair they should offer me some kind of compensation for the lost earnings, wasted time and lost opportunities.  They have declined this request leaving me entirely out of pocket for the 3 weeks I should have been working for them and, of course, unexpectedly out of work.

I’m obviously back looking for my next opportunity and I’m sure something will be along in due course.  This is a cautionary tale of what can go wrong in the world of contracting and, if your next contract involves TEKsystems or Allegis Group, you might wish to be extra careful, making sure they are actually able to offer you the work they say they are, and that you get paid.

Getting started with a UniFi Dream Machine Pro

It’s not an exaggeration to say that I’m an Ubiquiti fanboy. I like their kit a lot and my home network has been 100% UniFi for quite a few years now.

I’ve just moved in to a new home which I’m getting rewired and this will include putting structured network cabling in, terminating back to a patch panel in a rack in the loft. I have a small amount of “always on” kit and I wanted as much as it as reasonably possible to be in standard 19″ rack format. This is when I started looking at the Ubiquiti Dream Machine Pro to replace a combination of a UniFi CloudKey and Security Gateway, both excellent products in their own right.

My expectation was that I would connect the UDMP to some power, move the WAN RJ45 connection from the USG to the UDMP, fill in some credentials and (mostly) done! As I’m writing this down, you can probably guess it didn’t quite work out like that.

The UDMP completely failed to get an internet connection via all the supported methods applicable. PPPoE didn’t work, using a surrogate router via DHCP didn’t work, static configuration didn’t work. I reached out to the community forum and, in fairness, got very prompt assistance from a Ubiquiti employee.

I needed to upgrade the UDMP’s firmware before it would be able to run its “first setup” process, but updating the firmware via the GUI requires a working internet connection. It’s all a little bit chicken and egg. Instead, this is what you need to do:

  • Download the current UDMP firmware onto a laptop.
  • Reconfigure the laptop’s IP to be 192.168.1.2/24 and plug it in to any of the main 8 ethernet ports on the UDMP.
  • Use scp to copy the firmware to the UDMP using the default username of “root” with the password “ubnt”: 
    scp /path/to/fw.bin root@192.168.1.1:/mnt/data/fw.bin
  • SSH in to the UDMP and install the new firmware: 
    ubnt-upgrade /mnt/data/fw.bin

The UDMP should reboot onto the new firmware automatically. Perhaps because I’d been attempting so many variations of the setup procedure, after rebooting my UDMP was left in a errored state with messages like “This is taking a little longer..” and “UDM Pro is having an issue booting. Try to reboot or enter Recovery Mode”. To get round this I updated the firmware again, this time doing a factory reset:

ubnt-upgrade -c /mnt/data/fw.bin

The UDMP then rebooted again without error and I was able to complete the setup process normally.

It’s a bit unfortunate that UDMPs are shipping with essentially non-functional firmware, and it’s also unfortunate that the process for dealing with this is completely undocumented.

Buying a custom gaming PC from Overclockers UK

My current workstation and gaming PC is slowly disintegrating. I built it myself from components some 6 years ago and it’s simply wearing out.  Several USB ports don’t work and Windows sometimes bluescreens with errors that suggests bits of the motherboard are getting tired.  I don’t really have the spare time needed to build a high-end PC and make a great job of it, so I decided to treat myself to a pre-built custom system.   After hunting round, I settled on Overclockers as the company to buy from.

Their system configurator didn’t quite give me what I wanted, so I contacted them and asked if they could customise further which they could.   I put together my list of requirements, they send back a price.  I paid, cash wired to their bank account, upfront and sat back and waited for my new shiny liquid-cooled PC to arrive.

The system shipped.  It shipped to the wrong address.  I had provided Overclockers with a billing address and a shipping address.  They shipped to the billing address which is almost guaranteed to be unoccupied during regular working hours.

A simple mistake.  It happens.  I contacted the courier who were unable to redeliver again that day, but promised they would deliver it to the shipping address the next day.

Next day, my new shiny PC arrives.  I opened the smaller of the two boxes, one for spare components and so on, and immediately see a problem.  The spares and cables and whatnots are not branded with anything I specified, wrong motherboard and wrong graphics card.  I call Overclockers who suggest that the component boxes may have been mixed up and can I please open the main box and check. I do.  It’s someone else’s computer.  I later learn that my system has been shipped to somewhere else.  Overclockers’ mistake?  Courier’s mistake?  It doesn’t really matter. Overclockers have a courier come and pick up this system.

Meanwhile, my system makes its merry way back to Overclockers’ HQ and I, confusingly, get an email asking what I’d like done with it.   I suggest shipping it to the shipping address and could I please have an AM delivery so I don’t potentially waste a whole day.  I offered to pay for whatever that was going to cost.  Overclockers said it was no problem.  Super.

My PC finally showed up at Friday 8pm.   The more astute amongst you will spot that 8pm is not exactly an AM delivery.  Overclockers’ mistake?  Courier’s mistake?  I have no idea, the question has not yet been answered.

I unpack my new PC.  The first thing I notice is that there is a bolt rolling around in the bottom of it.  Stuff can come loose in shipping, so what.  I find that the bolt belongs to a radiator housing in the bottom of the case, there’s a hole, a loose radiator and tool marks around the hole.  Not ideal, but the system’s not going to be moved around much so no big problem.  Despite being an SLI system, there was no SLI cable installed linking the graphics cards.  Simple to fix, but a silly thing for an expert system builder to miss.

One of the customised things I asked for was the pre-cabling of some SATA drives bays: one for a blu-ray writer and two for a pair of big SATA disks I use for bulk local storage.   None of these were done.   I call Overclockers about this, and the loose bolt, and they say there’s not much that can be done without returning the system to them.  As I’ve no interest in another game of couriers, I grumble a bit but then do the cabling myself.

Over the next day or so I had almost no chance to really push the new system.  It ticked over happily, was lovely and quiet and lovely to look at too.  On Sunday night, though, the headphones went on, the office door was closed and I got on with a bit of GRID 2, with all the visual effects turned up to maximum.  I settled down for a couple of hours of hard racing.   After about an hour, the screen froze, went black, and all the system fans kicked into life.

I powered off, reached for my mini-torch and opened the case.  What I saw sickened me: liquid coolant leaking from the CPU block, down onto a graphics card and spilling on to the motherboard.  It was impossible to tell whether the CPU had simply thermally shut down or if the coolant had shorted something expensive.  It kind of didn’t matter.

The next morning I called Overclockers who arranged to pick the system up.  I asked if they could sort of the cabling and the loose bolts while they were at it.  They agreed.

A couple of days later, I got an email saying the system had been repaired and was on its way back to me.  The next evening I get a call from the owner of the billing address saying that a courier had tried to deliver something with my name on it.   They had shipped to the wrong address. Again.

I had now run out of patience and I asked for a full refund.   To their credit, Overclockers didn’t argue on this and they said one would be arranged.  As it was convenient for me, I asked to keep the Windows 8.1 licence and the SSD.   As it was convenient for them, I agreed to pay for these again separately, they would then issue a refund for the full amount of the original transaction.  I didn’t ask, but I kind of expected they would simply wire the cash back to my bank account.

After 3 days or so, nothing had showed up, so I called and they said that processing a refund might take up to 7 working days.

Today, 10 days on, nothing had showed up, so I called them and they said a cheque had been issued on the 4th and had been sent to……. you guessed it, the wrong address.  The owner of the address had not had a cheque arrive.

They offered to send a new cheque to the right address.  I suggested they simply wire the money to my account, I was told this was impossible due to the people who would have to do that being in Germany.  No, makes no sense to me either.  I asked if the cheque could be sent by special delivery, for which I was happy to cover the costs.   This was, of course, not possible.

So, 5 weeks after placing a cash order for a high-spec custom PC from Overclockers UK, I have no PC and they have a large amount of my money.

Please consider this post next time you’re thinking of ordering from them.

 

(Update: 15/4/2014:  A handwritten cheque arrived from Overclockers this morning. )

Communicado update: A change of tactic

The work to make Communicado’s life as difficult as possible continues and it does seem like we’re having some success.

When I started this project, Communicado registered all their domains through DAILY mostly using faked registrant data and hiding behind the privileges granted to individual private registrants.  I established a dialog with Nominet about this and it seems Nominet did take action to the point of suspending some of these domains.   Communicado then suddenly switched to using ENOM for registering their domains, I don’t know and have no way of knowing if they were booted off by DAILY or just decided to switch.  Either way, it made no difference, I could easily find the domains they were registering via Nominet’s PRSS tool.

As of Monday 16th, they have changed tactics again.  They have apparently abandoned the .co.uk namespace (I’m sure they’ll be missed) and have gone back to using a variety of .com, .net and .org domains.  Some seen in use today are:

actionallegiance.com
andronol.com
baotao.org
bigrockconsultants.com
coolpress.net
europacastno.com
greenroses.org
hourlycreative.com
pidchas.com

They’re easy enough to spot in the logs, but I don’t currently have a good way of searching the whois for these TLDs. Suggestions for such a tool (non-free is fine) are welcome.

Maintaining this list and the RBL service is taking time and money.  I will absolutely never be charging anyone for the list and the RBL will be free and open access for as long as it is sustainable to do so.  In addition to the ways you can help mentioned in previous posts, a more direct way you can help is to donate a little money, preferably in the form of Bitcoin to 1F9Y1Gd3Pmmchxa7uGFd3zBQY9zVuX78Jd.

More news when I have it, you can follow @Excommunicado for more frequent updates.

Unwanted email from Communicado Ltd

As my regular readers will know, I run an antispam and antivirus email filtering service called antibodyMX.  About two weeks ago, I started seeing some deeply weird junk traffic coming through.  A lot of it.  It was odder because it was going to an awful lot of different domains we filter.   Statistically we filter a very very tiny set of domains and to see junk mail arriving for many of them all at the same time was very strange indeed.

The domain names the mails were being sent from were also strange.  They were all .co.uk domains and all not-quite-words.   Here are a few:

bagumbayansa.co.uk
balabagansa.co.uk
balambanred.co.uk
balangasere.co.uk
balangigada.co.uk
balangkayansa.co.uk

All the domains are registered to an individual called Chris Hepworth, a company called Communicado, or “World trading Partners BVI 1611097”, or “Phil Neck”, or some combination thereof.  I called them up and asked them what was going on.  I was asked to send an email explaining the problem, I did, no reply.

More mail from yet more domains kept arriving, the domain count got up to over 300.

Using the resources of a somewhat underground entity known to some as The Fish Tank, I spoke to someone who suggested I sign up to Nominet’s PRSS tool which would let me search for domains more easily.  I did so and started domain hunting.

I fed the list of domains I had through a trigram analyser and used the results of that to tease more domains out of the search tool; “qua” was especially useful, finding more than 80 domains.

More mail from yet more domains kept arriving, the domain count got up to over 800.

Searching the web and talking to people, I found that an awful lot of people I knew were getting unwanted email from these people.  I also found one lady who is taking them to court.  I tried calling them again, my voicemail has not been returned.

Today I have spoken to the Information Commissioner’s Office who say they are very interested in the data I have collected.  It will be interesting to see what, if anything, they choose to pursue here.  On the face of it, Communicado do appear to be breaching the DPA.

I have been told there is no immediate reason why I cannot publish the list of domains I have collected, you can find it here and it currently contains 1255 3971 4500 4539 5145 5249 domains.  An example use might be an ACL on your mail server.  If you run exim, you would add an ACL along the lines of:

deny message = http://blog.hinterlands.org/2013/10/unwanted-email-from-communicado-ltd/
sender_domains = ${if exists{/etc/exim4/hepworth.txt}{/etc/exim4/hepworth.txt}}

The file will be updated as and when I have time.  If you want to capture it via a cronjob, you’re welcome to  but please:

  • DO email me to tell me you’re doing so, and from which host(s). You have permission when you click send.
  • DO tell relevant colleagues and friends about it.
  • DO consider donating a little money to charity.  Because of my nephew, I suggest here.
  • DON’T cron it for an obvious time like “12am and 12pm”. Spread the load, please.

You do, of course, use this list entirely at your own risk.

Kubuntu 12.10 and VMware Workstation 9.0 kernel panic

Yesterday I upgraded my laptop from Kubuntu 12.04LTS to 12.10 “Quantal Quetzal”.  One important change here is the move from Linux kernel 3.2 to 3.5.  The upgrade went smoothly enough but, upon reboot, I got a kernel panic from one of the vmware workstation modules when it loaded.

To fix, do the following:

After the upgrade, reboot to your 3.2 kernel (hold left shift down during boot to get the GRUB prompt), open a console session and then:

sudo chmod -x /etc/init.d/vmware*
sudo reboot

Let your system boot normally this time and you shouldn’t see the kernel panic.  Open another console session and then:

cd /tmp
wget http://communities.vmware.com/servlet/JiveServlet/download/2103172-94260/vmware9_kernel35_patch.tar.bz2
tar xfj  vmware9_kernel35_patch.tar.bz2
cd vmware9_kernel3.5_patch/
sudo ./patch-modules_3.5.0.sh 
sudo chmod +x /etc/init.d/vmware*
sudo vmware-modconfig --console --install-all
reboot

All done.

 

Running an oncall rota

Being part of an oncall rota is pretty much a certainty if you work in the systems or operations team at any IT orientated company. Stuff needs to be working 24 hours a day, 7 days a week and therefore you need someone available to apply duct tape and staples when things go wrong out of normal working hours. It continually amazes me, therefore, that so many companies get the management and organisation of their rota badly wrong, sometimes to the point of it being a significant factor in staff moving on.

As someone who’s participated in oncall rotas at all levels, here’s how I think one should be organised:

  • Sysadmins should only participate in the rota after they have completed their probationary period. Think of the goal of the probation period as being getting someone up to speed so they can participate.
  • Each oncall shift should last one week, and rotate at 2.00pm on Tuesday. If you run a late shift, i.e. 11.00am to 8.00pm, make the person on that shift also be oncall. Get all the unsociable hours out of the way in one lump.
  • The bare minimum gap between oncall shifts should be 5 weeks. If your rota is shorter than this, then you don’t have enough qualified staff to cope with holidays, illness, paternity/maternity leave etc anyway.
  • The oncall shift should be mapped at least 3 months into the future. Staff should be free to swap oncall weeks providing that no-one is ever oncall two weeks in a row, and that all swaps are cleared with their line manager.
  • Issue the oncaller with a good quality mobile phone and 3G capable laptop. The phone should not be a smartphone, the aim should be for maximum battery life and talk time. The laptop should be more like a netbook than a desktop replacement, should come with a spare battery, and should dual boot to Windows and a useful Linux desktop distro. These days I’d shell out for a nice big SSD to stick in it too. Make sure there are no restrictions on international calling and no data caps for the phone and 3G card.
  • Don’t give the direct number of the oncall phone. Instead, use an answering service as filter.  During office hours, the answering service should redirect enquiries to the regular helpdesk/support number.  Out of hours, the answering service should accept calls, take details and then pass these on to the oncall number. Additionally this service should text and email call details so there’s a record.
  • Pay a fixed daily amount for being oncall. Pay 1.5 times this amount for being oncall on weekend days. Pay 2 times this amount for being oncall on a public holiday. Where someone is oncall on a public holiday, add one day to their holiday allowance.
  • Pay a per-incident fee when oncall is used. Each oncall use should be tracked in your ticketing system. Make using oncall a business cost, thus giving the business a reason to make sure oncall is not used trivially, and a reason to make sure problems are fixed permanently and not just temporarily alleviated.
  • When the oncall person has dealt with out of hours issues, don’t expect them in at the regular time the next day.  Expect them to use their judgement to make sure they are suitably rested.   You do not want an overly tired oncaller dealing with problems on production systems.
  • The person oncall should never be taking on “out of hours” work. Want a database dumped and reloaded? Want a disk unmounted and fscked overnight when a server’s not busy? All those things can be done, but not by the oncall person. That person is there to respond to problems, not to perform routine or planned maintenance.
  • Make it very clear that abuse of oncall is unacceptable. Oncall is there to fix customer or service affecting problems, not to help someone with Excel.
  • Have a clear demarcation between production and testing/development/QA systems. The latter group are not oncall’s responsibility to fix.
  • If you have offices around the world, have a “follow the sun” oncall system, get your offices to cover each other.
  • Have realistic expectations of oncall response times. If you need to guarantee that problems are attended to within 20-30 minutes then you should be running an overnight shift, not oncall.
  • Expect a daily report summarising the previous 24 hour oncall period, even if that report is “Nothing to report”. The weekend period could be lumped together on the Monday.
  • Have a weekly oncall handover meeting between the outgoing and incoming oncall staff.
  • During the day, have a junior or trainee sysadmin be oncall. It’s good practice for them.

Doing all of the above shows that you take oncall seriously, and you appreciate the impact being oncall has on someone’s life. Your oncall staff are the people who salvage the business’s reputation when the midden hits the windmill at some unworldly hour of the night. Keeping them happy and making them feel valued and respected can only be to the business’s benefit.