18th December 2013
Recently we started receiving reports from a customer with our Managed Bonded Resilient Broadband services that they were experiencing reliability issues. This issue was particularly noticeable for the customer as they have reasonably long opening hours, working beyond traditional "office hours" to support the warehousing and dispatch team.
The complaint was that the warehouse team, based in Bristol were having trouble accessing the server systems in the customers office in Devon. Having opted sensibly for our resilient service knowing that a standard broadband line wasn't adequate for the job of maintaining robust communications, it was initially troubling to the client and indeed to us when we heard about it - after all our resilient service is designed to avoid the vast majority of cases where your broadband goes offline (although we can't do much about mass flooding, scenarios where diggers cut through cables and the like).
Fortunately, as the customer has our managed broadband services, we are able to help pinpoint the time and cause of issues, as we have direct access to the very granular and detailed information about line performance, availability and so on, helping us to troubleshoot the issue. This type of insight is often unavailable to small businesses as they work with companies who simply "pass the buck" of connectivity to other companies and don't provide the service themselves so it's a real advantage working with us when something like this does happen.
Our team quickly identified that the lines were operating perfectly well during the day, at a good speed for the area and (the customer is in a relatively remote location and has considerable distance to travel between them and the local BT Phone Exchange). There hadn't been any downturn in the sync/connection speed during the issues. It which suggests there isn't' a long term or sustained issue with the broadband lines in use.
Further analysis of the data showed that issues were only seen during the week, with service remaining stable between Saturday and Sunday consistently. During the week, issues only existed during the evening and only from around 17:30 to 19:00 on a typical day - with the odd exception where it would start a little earlier, or last a little longer.
Generally, the outages were very brief, because 95% of the time, the resilient service was handling the issue, with at least one broadband line remaining up giving uninterrupted service although we did continually see multiple lines struggle (our service is designed not to fail if a single wholesale network fails too so we could rule that out immediately). Occasionally though, the issue was so severe it did cause total loss of service on all lines - which is when the customer noticed it. Without the resilient service, they'd have been experiencing outages of 5-15 minutes several times an evening, but our service did everything it could to sustain a connection - leading to just a few seconds of outage. Better, but obviously not good enough.
With the data we had, we advised the customer that we believed they had a potentially complex (but surprisingly common) issue - either REIN or SHINE [see our FAQ for detailed explanations]. It isn't actually anything to do with the weather or climate although the name might suggest it to the casual observer, but a technical issue caused by interference (those with radio and electronics knowledge might well understand this issue already).
In summary, REIN and SHINE are events where other equipment is causing a type of interference or "noise" that prevents broadband from operating properly. Armed with the data, we advised the customer to think about things that they might have set to come on at a specific time - things like boiler timers, burglar alarms and so on - and even things like christmas lights (especially the flicking/flashing types). They had a look but couldn't be sure of anything - we also asked them to chat to a neighbouring business to see if they either had issues or had equipment coming on/off at those times. It makes sense because the lines pass through both premises to get to our customers.
Having exhausted on ideas, and no real luck in finding the culprit, our engineer was asked to have a look whilst on site working on other projects for them as part of our Managed IT Services. They noticed that the customer had a security light which was being used (one of the types with sensors). Suspect that this is to blame in some way, given the timings, he asked the customer to disable/turn the light off so it couldn't activate anymore. It was potentially a long shot, but the most likely cause. In discussion, it also transpired that our customer had staff leaving at varying times starting from when we first saw the issues - the days with the later failures were those where senior staff were working late.
The security light has remained disabled and the broadband service is now robust again - even though our resilient service does its absolute best to keep your connection live, it can't beat careless contractors digging through cables nor the laws of physics!
If your business is struggling with reliable broadband, our team can help you troubleshoot the issues, or even provide our managed and/or managed resilient services.
Are you getting the best out of your IT?Take the Quiz