February 10th, 2022
Backups and Testing Backups
There are a lot of sideways looks thrown around when we talk about things like backing up Exchange Online mailboxes (part of Microsoft 365). In this scenario, everyone understands that Microsoft has provided a recoverable and resilient infrastructure for the email services and the mailboxes they are hosting in their cloud, as well as other 365 and Azure based services. So why should be care about backups, after all Microsoft has our backs…right?
Nope. Microsoft does not back up customer data. To prove that point, there are plenty of third-party solutions, and even Microsoft has their own solution for performing backups of customer data, including email data, at an additional cost.
Generally speaking, the discussion above almost always turns to, “who cares about email?” Comments are made that email, “is not the sword that the institution dies on,” or, “it’s just not that important.” Or even ones like, “Microsoft is too big to fail.”
Granted, the likelihood of something catastrophic or that Microsoft would suddenly become insolvent, is extremely low. However, the unknown vulnerability still exists, and Microsoft, just like all other entities connected to the internet, is at the same risk as everyone else. They just have more money and resources to throw at lowering the inherent risk.
In all likelihood, your institution has—or should have—continuity plans in place based on proper risk assessment to temporarily compensate for something like email failure. But the inherent risk of relying solely on Microsoft to mitigate said risk is huge. And without a plan in place, like tested backups of email data and potentially other gateway-based resiliency solutions, the residual risk is the same as the inherent risk.
Here's a challenge: ask your agriculture lenders how important their email is for moving documents and letting farmer John know his lender paperwork for that new array of low-oxygen tower silos is ready to electronically sign. After all, farmers are getting more tech-savvy, just ask one that’s tried to fix their modern axial-flow combine without the help of the dealer and licensed software. The same surely applies to your mortgage originators and their communications with home buyers in what has been a very busy housing market.
The point is some institutions haven’t asked or taken the time to ferret out just how important something like email really is to their organization. And it wasn’t really of high concern during the migration of on-premises Exchange to the 365 cloud, and quite possibly wasn’t even broached by the MSP. Our audits and security assessments across hundreds of clients annually paint a very telling picture of situations just like those.
So, there are seemingly two choices in this context. Either eliminate email entirely or find a way to back it up to meet regulatory standards and examiner expectations. I’d venture to say that your average daily customer interactions beyond the teller line won’t allow for the first option.
To further illustrate the point of the dangers of solely relying on the MSP, take Kyoto University and their significant loss of research data—77 terabytes to be exact—as an example. It was an awfully bad day for poorly managed or missing backups. Based on the limited report from Kyoto U. and other news from media outlets, this backup failure was due to a poorly designed backup schema where the previous backup job was immediately overwritten by the latest backup job. In other words, there was no incremental backup data, and no retention of backups. A failure or corruption of data necessitated a restore, at which time it was found that backups were not available.
Furthermore, it was reported that Kyoto University is blaming Hewlett-Packard Japan, G.K., (which is no small company, like Microsoft, and our aforementioned MSP example in this case) for the mismanagement that led to the backup failure and eventual data loss. Data loss that affected fourteen research departments and their research projects at one single institution on a shared, leased super computer array, not too dissimilar from the cloud concept that Microsoft relies on, and likely similar to how your core provider manages your iSeries system, or your document imaging system, or your mortgage processing solution. After all, the cloud is just somebody else’s hardware, right?
It can be surmised based on the limited evidence and reports that Kyoto U. did not follow through with reviewing and monitoring contractual obligations of the MSP to test and validate that backups can be used for recovery, presuming they existed in the first place, which apparently they did not in this case. Furthermore, one might come to the conclusion that this incident was a result of a failure of to manage risk, continuity, policy, vendor relationships, and infrastructure, to name a few issues. And ultimately a failure of management to maintain oversight.
Have you tested your backups to know that they are any good, or if the backup solution is confidential, has integrity, and is resilient? And, if you are paying a third-party managed service provider to perform those backup tasks, can they prove they are doing what they are supposed to do?
To end this, I will leave you with an anecdote. At eight years of age, I cut my teeth on a used Apple IIe my Dad bought me, and an Apple IIgs signed by none other than Steve Wozniak alongside an IBM 8088 that I learned things like DOS and BASIC on in my Dad’s office. We even had a “modern” track-fed teletype machine we used as a printer, that I would occasionally use to dial up the BBS at our local community college and watch it print out the ASCII welcome page and menu that I could interact with via the teletype’s keyboard, just for fun. Above the 8086 monitor hung a plaque: “To err is human, but it really takes a computer to foul things up.” That statement is so true, until you can prove that it’s not. Thank you, Dad.
Authored by: Mike Smith, AWS CCP
Download Blog
