Behind the Scenes @ VPW - Backups...

26th August 2014

From time to time we've been asked in the past what we actually do here in Exeter at VPW Systems - all the things that our customers probably won't know we do but which ensures things stay running and the services we provide.

This weekend we've been working on an extensive upgrade of our Backup Systems - those that backup our core infrastructure - the servers powering our services such as Agility Mail & Desktop, Web Hosting and so on...

So what have we done?

Last week was spent building and testing new servers ready for deployment - because we store pretty large volumes of data (many many terabytes!), there's no practical way for us to buy pre-built systems with the kind of storage setup we need at a cost that isn't so utterly eye-watering it hurts (not to mention that we'd also then need to spend hundreds of thousands of pounds - literally!). That's mostly because typical servers aren't really designed for the purpose and way our backup systems work. We're not alone and plenty of other "high storage capacity" businesses build storage themselves - there are a few notable examples in the United States for example.

Backups are a serious matter

We take backups and protection seriously, so we have 2 sites where backups are held - the first is at each data centre where the servers themselves live - so each geographic location and data centre has a local backup storage server (or server(s)). This is useful for the speed and ease of of backups, makes restores for "accidentally deleted files" etc simple. However this wouldn't help if a fire hit the data centre, or theft (it does happen!) or even some sort of natural disaster.

To make sure we're safe, all the backups and protection then lives also at another location too - known as a secondary protection zone. In the event of a loss of our primary data centre for a group of servers, we can get data restored elsewhere and get back up and running.

In this case though, we used this capability so that we could first divert new backups to the secondary location, because the primary location would be offline for a day or two - allowing backups to continue as normal - after all, if you didn't do this, you'd have no backup at all and the data we store for customers changes constantly.

This done and tested, we moved on.

The weekend upgrades...

The next step was to physically rebuild/replace the backup storage systems at our facilities - this meant engineers from our team attending, removing the old systems, backing up those systems and the databases that track backups, then completing the build of the new servers on-site - including adding the storage capacity they need. From there they'd need the production operating systems installing (our core Infrastructure is largely Windows Server 2012 R2 based, so we're right up to date on technology - this brings other interesting challenges), the backup technology and software installing, and then the status databases restoring. This process took around 12 hours per storage system. Once ready, they're physically placed into the racks and final configuration completes.

In full the process included:

Backing up the Existing Systems (yes, even backups get backed up!)
Physically building new Hardware
Removing old Hardware
Installing Operating Systems, Drivers, Patches and Security Updates
Installing Backup Systems, Restoring the Databases for them
Deploying them and testing basic functionality is ready to go

Switching Back to Normal...

With the new systems in, the next steps are to switch protection back to the primary systems from the secondary protection - effectively reversing our earlier changes - there's no reason you couldn't leave it the other way around if you wanted, but it makes sense for the most local backups to be considered "primary". This process takes a while as current backups are completed (bearing in mind our systems do many backups every day - not just overnight backups), and things are handed back.

Monitoring & Documenting...

Finally, the current status for the next 2-3 days is to monitor everything, making sure all backups have handed back, that they're running from the primary protection again, and that the secondary protection zone goes back to having the additional copies just in case. Assuming this completes, our upgrade will be completed from a technical perspective, leaving some documentation to be written up, procedures revised and asset information and other miscellaneous data (including photos) that ensures we have comprehensive idea of our infrastructure and systems down to every last cable.

If you'd like to talk to our team and find out how we can help your Small Business break away from the troubles of IT Services and would like a competent, capable team to take care of it for you, why not speak to us today on 01392 950 950 and we'd be more than happy to help you. After all, we're here to make IT easier for you!

<< Back to the Blog