Who’s Got the Keys?

Your institution builds a new facility in a high-crime area. This new building will be used for corporate operations overflow. There will be executive offices, lots of IT infrastructure, and rooms full of sensitive paper document archives. With consideration of crime, exterior windows are protected with bars and have break-glass sensors that tie into the centralized security alarm system. Entries and exits to the building are covered by surveillance cameras and all the usual detective controls.

An after-hours intrusion occurs. Someone left the main entrance unlocked overnight.

Management scrambles. This cannot happen again. Policies are implemented, and staff are trained. It’s also decided that the employee entrance will only be used as an emergency exit going forward. A security consultant is engaged to assess the physical environment, and many useful recommendations are considered. Among new controls that are implemented, a security guard firm is hired to attend the entrance during all hours that employees are present. As part of their operating model, the security guard firm expects to be the sole holder of keys and codes to unlock and enter the building.

What happens next? Would your management be comfortable with this arrangement? Would you give up control of the only set of keys to your front door?

We see this scenario play out more often than you may think, sort of; Replace “high-crime area” with the Internet, replace “door” with firewall, and replace “security guard” with managed service providers (MSPs). The Internet is a scary place, and if you have access to it from your corporate network, your network is a part of the Internet. Protecting your internal assets from public access is essential. Layered controls on workstations and servers and in-line network detection and response mechanisms are more advanced today and have become more popular than ever, but the first line of defense for most private networks has been the same for decades: the perimeter firewall.

Firewall management has become easier since the days of exclusively command line configuration changes, but there’s still a need for fundamental understanding before a staff member is allowed to add new access rules, remove existing rules, change content filtering settings, or setup logging controls. Many opt to hire experts for this purpose, and MSPs are often a good fit for firewall administration and other network equipment management functions.

We commonly encounter clients that own all their network infrastructure and outsource management to third parties. Through deeper dives into these arrangements during audits and assessments, we’ve discovered many of these relationships have been established with only one set of keys. Sometimes dictated as a requirement in MSP contracts and more often through informal processes, organizations sometimes allow MSPs to hold the only administrative access account for assets such as firewalls, Active Directory, phone systems, and other critical infrastructure, especially when there’s no IT staff in-house to manage them.

There are a few reasons that the practice of relinquishing administrative access to MSPs seldom makes its way into risk assessments. One reason is that the vendor management team thinks that IT is considering the risks, and at the same time, IT thinks that vendor management is considering the risk. Where does the responsibility lie? To answer the question, both. Both areas should consider risks of a single external entity having logical control over things like the firewall, but the most popular reason that this practice seldom makes its way into risk assessments is that there’s not been an actual instance of it being a problem.

What would you do if an MSP employee went rogue and made a malicious change to your firewall configuration, like creating a backdoor VPN session that they could use to access your network? What if that rogue employee was a partner or owner of the MSP?

What would you do if your MSP became non-responsive to change requests or stopped answering their phones? What if you heard about a security incident at the MSP from another client, on social media, or on the nightly news? What if the company were to shut down suddenly for financial reasons, skilled staffing troubles, or law enforcement action?

If such an arrangement,one set of keys, applies to your environment, I’m sure many responses will come to mind.

Some of those may be in the realm of ignoring or accepting risk: We monitor their financials. They’re trusted partners. It’s never been a problem and is highly unlikely to ever be an issue.

Some may be reactive: We have insurance. They have insurance. We keep firewall backups. We’ll contract one of their former employees directly. We’ll get a new MSP to rebuild the firewall configuration.

Those common reactive responses are still heavily hinged upon assumptions. Will costs caused by an MSP disruption qualify for insurance claims? Does your contract with the MSP contain hold harmless clauses exempting liability? Are firewall backups current or from a year ago? How long will regaining access to the firewall take, and can it be done during the day without disruption? Will their former employee(s) agree to help us? Will a new MSP set us up for this scenario all over again, and do we have time to wait for establishing the new relationship? Who’s handling this recovery if the MSP was named as a lead member of the incident response team?

Rather than focus on a reactive response, let’s offer an easier and much more reasonable, proactive, and preventative solution. If you own it, keep a spare key - just as you would for a house or car that you own. Request that your MSP create a fully functional backup admin account for all systems they manage on your behalf, and if your contract with the provider stipulates that they are the sole keyholder, factor that into vendor management assessments and selection criteria. Keep backup credentials locked away in either a physical or virtual vault. Test them periodically for general functionality. Ensure relevant management is aware of where these credentials can be found in case of emergency. Lastly, ensure that your annual IT audit program evaluates administrative access for all critical systems.

Authored By: Kyle Stelly, CISSP, PCIP

  Download Blog