Sky Lakes Medical Center, a 176-bed community hospital in Klamath Falls, Oregon, was one of the major health care providers affected by the wave of cyberattacks in fall 2020, confirmed to be Ryuk ransomware. Sky Lakes responded quickly, working with their Epic Community Connect host, Asante, a health system serving southern Oregon and Northern California. Community Connect is a program that allows hospitals to connect to a larger hospital's Epic EHR network. After the ransomware attack on Oct. 26, 2020, Sky Lakes operated under downtime procedures until the hospital was cleared, almost three weeks later.
Advisory Board's Mark Hetz, Senior Research Director, spoke with Asante's CIO Dr. Lee Milligan and Sky Lakes' Director of Information Systems John Gaede to discuss the unique role their Community Connect relationship played in responding to the cyberattack and lessons learned.
This interview has been edited for length and clarity.
Question: How did you first learn of the ransom attack?
John Gaede: I was called at 3:30 am PST after the encryption notice was discovered. There was a complaint about system slowness at 1:30am, about 13 hours after the attack, which is problematic.
Dr. Lee Milligan: I was at home eating my breakfast when I got a phone call from John Gaede. He let me know that Sky Lakes was infected, and that his team believed it was the Ryuk virus. They were still in scrambling mode. It was apparent that hospital officials learned about it around 1:00 am PST and I learned about it at 7:27 am. My first concern was the gap in between when we knew.
Q: What was the first thing you did in response?
Gaede: We shut all systems off to protect against further infections and decrease lateral movement. We also contacted Asante leadership so they could disconnect.
Milligan: I pulled my team together to mitigate. Our concern was exposing our other Community Connect sites. We basically shut off all contact between us and Sky Lakes. We shut down the virtual private network, Citrix systems, file transfers, and all emails coming from Sky Lakes. One of our key decisions was to take backups offline because we decided it was better to miss a couple of days than to risk everyone. We didn't bring anything back online until we were sure there wasn't widespread exposure.
Q: What were your next steps? How did the response evolve over time?
Gaede: We had to take all Windows-based systems down and disconnect from the network. We had to call each vendor to ask what they'd recommend and to see if their operating system could have been compromised. The saving grace was that our backup was on a Linux-based system. Once we validated that each system was clean, we restored to the network with full patching and clean scans.
We had to reset 100% of password accounts for all users. Now, we match Asante's requirements for passwords and require a minimum of 13 characters.
Finally, we had to agree to Asante's new Memorandum of Understanding (MOU). We needed the document to get Epic back up and running securely. Before we could go online, Asante also asked for a go-live plan. Once we did that, we restored Epic to full clinical use enterprise-wide and restored third party systems to support Epic. We then did house-wide reconciliation, calling in nurse and pharmacy resources to do back entry of all records.
Milligan: You know, we really have a lot of great things in place. For a smaller health system, we really had a comprehensive approach.
We did some things to ensure our scenario was secure and that the attack wouldn't move laterally. We had a tool to confine the attack, but unless it's configured appropriately, the tool doesn't work very well.
Most importantly, I didn't want to reconnect with Sky Lakes until it was clear to me that the hospital had created a fully clear system. I put together a MOU that the hospital had to sign before we could reconnect. It was fairly descriptive—I wanted to provide a framework for John to take back to his team to ensure our safety and act as ammunition for his conversations [with senior leaders]. A couple of things we included in the MOU:
- Sky Lakes needs to have a reputable third party come in and give a clean bill of health.
- Sky Lakes needs to inform Asante within 30 minutes if they've been breached.
- Sky Lakes has to have a yearly security evaluation, penetration testing, and results.
- Sky Lakes needs to follow NIST, v2 information security framework
Q: What resources (time, money, people) were required?
Gaede: We were down for three weeks, so we lost revenue. The attack happened on Oct. 26, and we didn't drop a bill until Dec. 28.
We also had staff working nonstop, nights and weekends, so we had to pay time-and-half to hourly workers.
Milligan: We're currently calculating that. We met with Sky Lakes daily and offered best practices, so there were a lot of hours spent internally and externally supporting them.
Q: What was the hardest part about responding to the attack? Why was that so hard?
Gaede: The process of validating that each system is clean and restoring them to the network is more time consuming than anyone talks about. It will take us an estimated five months of hard work to restore all systems.
Milligan: It's the unknown. You think you have a good posture in place but then something happens, and you get paranoid. Once we got to the spot where we were reconnecting our backups, we felt good.
Q: What would you do differently if you had to do it all over again?
Gaede: We would negotiate with a cybersecurity insurance provider earlier to know what tools and resources to bring in. Time is of the essence so having that pre-negotiation would have been great. We had to do a lot of work because we did not know how lateral the infection went, so we had to assume everything was compromised. We had to reimage or replace every device in the organization.
We also didn't have our endpoint management software fully configured for blocking the execution of this malware as we were in transition from one vendor to another. The software saw the attack but didn't block it because it wasn't fully configured. Now we will ensure we have a professional services as part of the configuration process to ensure fully configured protection.
Milligan: I think in hindsight, these conversations should have happened before the fact. John and I should have had an understanding that Sky Lakes would let us know if it was breached.
Q: How has this affected the relationship between Asante and Sky Lakes going forward?
Gaede: We had a stressful time especially while waiting for the MOU before we could reconnect to Epic. Neither organization had experienced anything like this prior. Understanding what was required in the MOU influenced our prioritization in the restoration process. In retrospect, the time spent crafting a meaningful MOU impacted our ability to safely re-connect. I'd say to Community Connect organizations across the country that it would be helpful if hosts had something ready beforehand, which could potentially save a couple of days and translate to saving time and money. In the end, the Asante team came up with a reasonable document that added additional levels of security and value to our relationship.
Milligan: It's interesting. On one hand, there's a lot of thinking through the nature of the relationship and what it means to our organization and Southern Oregon to have all of us under one system. In general, it's a really good thing. But we realized that this connection does pose an ongoing risk and we have to understand that. So, we're stepping back and looking at it from a business-risk perspective.
I do want to point out that the Sky Lakes team did an incredible job of responding to this attack given the circumstances. Led by John Gaede, the team rallied their troops in a fashion that can only happen with a strong leadership and great team collaboration. They were wounded but not down for the count and they ultimately came through and delivered for their community. Hats off.
As the Community Connect host, Asante did everything it could to support them—sharing best practices, meeting regularly to review the planned response, identifying standard work, and strategizing on next steps, including the re-start of Epic. I do feel the teams became closer as a result of this experience.
Q: Has this event changed the organization's focus on cybersecurity?
Gaede: Yes, especially as the incident is fresh on our minds. The organization is receptive to technologies that support security. However, the reality—weeks of lost revenue and Covid-19 pausing procedures— will make for interesting and challenging conversations. The resources may not be there.
Milligan: Not much has changed in the short term. That's primarily because we had built up our cybersecurity pretty good. In September, we did a table exercise around ransomware, not unlike what happened to Sky Lakes. It was a really nice exercise to help us think through it.
Going into fiscal year 2022, I'll ask for more resources, but not right now.
Q: What was the biggest lesson learned?
Gaede: A few lessons learned:
- The importance of good backups—we would have had to put a for-sale sign up if we didn't have them or if they'd been compromised.
- Have a good Security Operations Center (SOC) to monitor 24/7/365. Most hospitals don't have an SOC even though it's the number one recommendation from most information security experts.
- The first line of defense is education. Educate employees about not clicking on email links!
- Well-thought-through downtime paper processes are a must. Downtime for a long period is the worse-case scenario. Even when we got back online, we needed those paper records—both on the financial and clinical side to do trend analyses.
Milligan: Security is like a utility—people expect it to happen until it doesn't. You want to get the message out without being perceived like Chicken Little.
The best way is to put it in terms of business risk—what are we doing, what will happen if we don't, and examples of what happens when it goes wrong. I'm not trying to scare people, but I need to identify real and ongoing consistent threats.
Q: What advice would you give to others who go through something similar?
Gaede: My advice for Community Connects:
- Investment in cybersecurity is a must. A lot of hospitals aren't investing like they need to. This includes investing in legacy systems, like imaging systems. You need to get off legacy systems to reduce footprint. The bad actors didn't gain credentials from PCs—they got into a domain controller from one of the legacy systems. You also need Managed Detection Response, a service to watch your network and flag issues for you. Also, specifically for community hospitals, making investments in a redundant data center. It's costly, but if we had a second data center, it's possible that we could have moved systems online quicker.
- Have a playbook. People talk about playbooks, but it's not enough to have a policy in a binder somewhere. We want our playbook to actually guide us. We are shifting one of our employee's job to creating a downtime process for everything from environmental services to food services to nursing. We weren't ready for 23 days offline. We can't tell you the extent of revenue we lost due to paper-based documentation that was not completed properly.
- While we were down, we upgraded everything. So, the silver lining in all of this is that we no longer have Windows 7 in our environment. Everything was updated to Windows 10 while we rebuilt or replaced all 2,600 PCs.
Milligan: I'd think about the framework of Community Connect in terms of security. I'd pull out contract and identify SLAs around when Community Connect site has to let host know about infection.
We have since established a quarterly meeting with our security team and the Sky Lakes security team so we can support them and keep accountability with our mutually agreed upon MOU. We want to make sure both systems are framed up for success on the frontend so we're comfortable that each side is taking all the right steps.