Lessons Learned on Cybersecurity from the EHR Vendor’s Perspective

[Moderator] Chris Alban's going to talk about lessons learned on cybersecurity from the EHR Vendor's Perspective.

[Chris] So if we were running short of time, I would just use this slide to summarize. And I'll back up. So I'm Chris Alban and I'm one of the physicians on the surgical team at Epic, on background medicine. I had the pleasure of being a part of this section. And for Epic sponsoring us for a couple of years now. As a vendor you get a few minutes to talk about something. I'm trying to keep it relevant and knowing that Christian was going to come up and talk about some really cool stuff in cybersecurity. I thought I'll uh. Ben said, "What you want to talk about?". I thought well, maybe just to a review of something we've all learned at Epic on cybersecurity problems. What if we've got a lot of customers, we've got some big folks around the country. Some have, unfortunately had some very troubling experiences over the last several years. And so what have we learned as a vendor who's supporting them try and help through that. And a few things start to come through and I thought, you know, that's at least a little bit of a segue into, again, Christian doing the really cool stuff and talking about this in a way that he only did. This in a nutshell, is that he's lying. Protect, if you look at the verbs. Protect, prepare and prepare, and if anything comes through over and over again anything we've learned with our customers, is those if you don't do it in advance, you're going to get, it's worse. So 60% of a good plan now is better than 90% later, do something. Whether you're at a small organization, a large organization, you're with a vendor. Help in any way on planning ahead makes, it makes a huge difference. So the prepare is the, what can you do in advance to protect? And a lot of this falls to the back end high technical side. As docs, as End Users, as frontline folks. It's the be careful of what you click on in your email because you might be the one to send it through. So it's fascinating as we look at the, the landscape of what's happening it's phishing schemes and social engineering. They're looking for any vector they can get in. And they're constantly going at it they're increasingly sophisticated, it's a business. I heard a phrase and maybe Christian can expand this ransomware as a service. I mean, it's financial thing and they look at it that way. They're sophisticated. The bigger the organization is. Because that's where the money is. It's big organizations, because they have more opportunity, more money to spend. So the bigger the organization you're a part of.

[Man] Rather keep your eyes open

[Chris] the bigger, the threat you, or the bigger opportunity you might present to them. And once they get in, they're in contact they're going to keep digging once they're in. Once they stealth after activity and they will do everything they can to make it happen. So again, and this is this from Microsoft. I love some of these. I've never heard of them. Maybe Christian has, the EV cats Paul ball strikes with some great games for this horrible stuff and the threats they present. There's lots of resources out there, whether it's Microsoft, whether it's all vendor sites, whether it's or our sites, we've got a lot of it. Every one of them is different. Probably the second bullet point is probably the most important point of this. And I've heard it from Ray Keller. Who is in the informatics section from the university of Vermont, and they provided us with some of the background here cause they presented that to us. But he, the adages are not measured in hours. This isn't your standard technical down time. These are days to weeks. And they said, Greg don't. When someone says, "how long is this gonna be?". Don't say anything. And he said, he didn't follow his own advice, he kept saying, Oh it can't be that long and 26 days later, finding it go back up. It's days and weeks, it's not hours to get back. And it's, so much of it falls into the recovery range. We'll talk a little, just a little bit about some of that. So what do you do in the prevention realm? There's our checklist here. There's a checklist all over the place in terms of that sort of stuff knew. It's email protection, it's multifactor authentication. Security administrative access on and on all of those. Some of 'em technical. It's educating your End Users, don't do dumb things, don't

[Indistinct] stuff you don't know. We have an where they're always sending on fake things to see who's susceptible. So they know who to do additional education on. So it's education and awareness becomes huge parts of this. And none of this is certainly from my perspective, advocating. This to everybody. Everybody who's inspection now. Both for external resources, vendor, and otherwise for more on that, I hope the other vendors do this. We do a security update for all of our customers to show where you are. Have you taken all your updates? This is important with Microsoft and Apple. Take the security updates as soon as you can. Stay as up to date as as possible, because that's what makes a difference. They're working hard on this, as well as your technical partners in a lot of this. Let's look over some of this. Remember your portals. Remember your mobile devices. Remember the page portal. Remember all the different things that can be affect. All right. So the really fun part happens when it actually happens. As the identification, this is the part where you can be prepared and can actually relate to you. If you're in the ED, what are you gonna do? And a lot of it comes down to planning. What are you gonna, what happens to your workflows? What are you gonna do when it happens? And a lot of this is also have the plan in advance back to that CIO's quote. If you have some of that there, then at least you can react. If you're gonna engage with an incident vendor. Do that before this ever happens. You don't want your vendor selection when the strike happens. So again, in advance of this new service. Recovery becomes a real talent part, and especially for the clinical operations. Reach out to your vendor. Foster whoever it is for help. Again, this is days or weeks not hours. They're all fairly unique. And their recovery becomes part of that real challenges I would say. There's a range of your access. You can make you down to just your business conopodium access PCs with no connection. The email's gone, Internet's gone. The only thing left is your guest internet. And, and you're basically using cell phones to communicate because that's all you got left on. And let's hope your cell signals good and you're, because otherwise depending on your own guest internet, to be able to communicate. Because communication's gonna be part of it. And as time goes on, you'll phase it to more access to your HR and other systems to get back up. So does the staff know what the downtime procedure are? Are your forms up to date? Have you review them? Do you have a regular cycle for reviewing? Does your staff know how to write paper orders and how to write paper prescriptions? No, they don't. You guys, some of you guys are dumb enough. Na mark, I have here on paper. You can hope the exact same. All right, so, key questions for your staff. Are you old enough? Oh, first of off and have you learned, and have you done those processes to know how to do those pieces. and then how does that P already get to the where it needs to be? What's the process you want figuring that out when this happens. And your back field question becomes a larger strategy of what is gonna be in that, and that. You need to think about the advance. So lots of technical recovery pieces, a lot of that last one you want to know in advance in clinical back field, what is your level of that. This applies to regular downtown, as well as nasty multi week cybersecurity, ransomware type ones. What's that level gonna be? when do you do it and how do you do it. It's not just clinic staffing models for first of all. For UVM first month this sort of summarizes it in some ways Maybe just bring everything back up. There was a staggered phase of bringing up their non EHR systems. And so one of the question was how do you it's like going live in a phase way. You introduce hybrid workflows and hybrid challenges and continue those change. So the End User are continually having to deal with the new set of workflows and new set communication procedures. Communicating that is hard. Getting through that is, is difficult. And so for example, we can certainly identify what that would be. I can prevent dead end orders. How would they get to where it needs to go radiology lab, other services? Do we actually have a person who runs that piece of paper to radiology so they know we need to new a chest. You gotta have that kind of stuff figured out because that sort of introduces a whole safety layer of monitoring to make sure we're not losing patients, causing harm issues. Just through not even being aware of what's going on. All right. Support stuff, have a plan, prepare in advance. Keep calm because we have a plan. Lots of those checklists. It's a lot like, one our customer equated into a recipe. There's lots of recipes for this. How you bake that cake will vary but it needs to be specific. You need to know if that recipe is in advance as you proceed through.

[Man] Hi, I'm at a designated part question asker. All right, so, by most accounts were down to three major EHR vendors. A minor or what is those risk of more about that? So the entire Epic, environment, going down. So you have multiple, you know, implementation of your DHR but is there a way in which listening that someone could bring down Epic enterprise across the whole, communicate?

[Chris] I think the short answer is no because they're not all connected. So MENGOS is an example, is hosted by Epic in fast recovery in multiple locations. But NGB partners, they're on their own. So they own their own data center as does say Lehigh, University of Maine and a bunch of others. So while we have a chunk that's in our own server bunker, card and everything else. I suppose that has a risk. But we have a whole team that monitors that it keeps at a very much protected. And with all the approaches we possibly do. Working with Microsoft external ones would be separate. [Man] I thought that that the my Epic thing was cross organizational. Sort of bring all data into an individual that, is that not right?

[Chris] The, the so care everywhere in interoperability is a point to point change at the point of care of Clinical Mayo. So if I present it to Clinical Mayo they could at that point, cause it's a clinical encounter, pull my stuff from Madison customers for that can. So it's not all centralized. It's only pulled in that sense immediately. There is cosmos, which is our data center or data analytics compile that for customers, but that's embedded. That's not for clinical care. That's only a identify and go find research data.

[Man] So it's connected in that way. Care everywhere is, has some connection.

[Chris] Yeah.

[Chris] Yeah, yeah, no, it's not a day of cost disease. So I guess perhaps to we'll accept the question could the carers data stream be corrupted to allow entry? I don't know. I I'm sure. We, I don't think so. Cause it's really a straight data stream. And my understanding is no one's actual Epic data base has been probably point. All of the attacks have been on me otherwise on networks bringing out their virtual machines, all their network pieces. So the challenge has always been, how do we get at least get the Epic pieces back up clinical operations? Cause there, there have been, they've been okay at that It's just that everything else get brought at the same time.

[Man] There's a little bit, it just begins to concern me as we begin to connect systems that we make the vulnerable widespread attack in one hospital. Obviously, could be a disaster. But if all the hospitals in California, you know somehow got taken down. That would be, you know, catastrophic. So let's generalize the question. Yes. We have a big responsibility in this as a vendor to make sure that the more, the larger our spread and this goes even internationally. There's a whole bunch in Ontario, in Alberta. You know, there's a lot of them in Australia. The same. We have a big responsibility as a vendor and as a certain all script Cheddite. Any of those to be a key partner and to make sure our stuff is as hardened as possible and that we do everything we can. It's, that's our responsibility. Cause this is day to day operations of, of, you know million of patient encounters, clinical ed inpatient and on and on and on so.

[Man] Yeah, and even the department is this.

[Chris] There's no perfect answer. We have to do all of these things to the best that we can. So is there potential risk? Yes. What do we need to do? We have a team, we keep focusing on it. As close to this as we can.

[Man] Maybe related, but Epic reliance, still a lot on Windows and on Citrix for access. like I wonder does Epic partner with Microsoft or Citrix about security because yeah.

[Chris] We, we partner with Microsoft decades of staying very close to them in our, in our basic architectural levels. We have very close connection, on a lot of that. So we, we stay very up to date on what we need from them, in terms of maintaining secure networks and, and secure operations. We do the same with Citrix and that's, those are probably the two biggest ones.

[Man] Yeah. When I, when I tell when I do some IT troubleshooting and, "oh, you don't actually have an Epic problem you have a Citrix problem", like that is not as comforting as. End User doesn't seem to make the decision.

[Chris] And that's, that's a big point maybe this person will talk about this as well, whether they bring down your network. They may not be actually releasing or extracting the PHI but it doesn't matter to the user if I can't use the system to take care of our patients. We can't operate our EV, we can't do our operating room, we can't see patients in the clinic. So you're right. Every place gets affected, even if they brought down your network and no PHI gets exposed. So. It's a valid point that there are multiple points of failure in there that all need to be working in order to get this security. He's just, Chris just. he's gonna color this in with much more depth.

[Man] Sorry. I have an Epic related question too though because since I'm a Epic End User my understanding is that there is a plan and an active transition to more of a web-based interface.

[Chris] Sure.

[Man] Is what are the security implications there? You don't have to know the answer. It seems like they're new and diverse and different. I plan, I'm preparing Chris, I'm preparing.

[Chris] So, so there is a plan obviously. We have actually a, part of our standard UGM and XGM user meeting. One of them is a security architecture, focus grouping and appraisal, a lot of chief security officers and all the folks on it who would be on an annual basis. So we continue to push the envelope on how do we create, say a shadow server that is rolled off so that if this happens you've got a backup that has been protected. And there's a number of those types of techniques as well as many others that our security team works on. I hope the other vendors are doing that as well because it's, you know, all of us don't want patients harm and, and, and why people can be able to do their jobs, exactly.

[Man] One thing I would add to is that on the web development, a bigger percentage of our time is what are the, from itself. And so we do a lot of the web server. How can we validate the input and make sure best security point, right? So it just sort actually comes with the webs technology security protocol

[Man] And exactly on that point. I'm actually rather excited about a browser base, sorry just on my point, pretty excited about the browser base. As long as the browser selected was a, a focus of scrutiny. One of the biggest problems is that we fill these frame inside systems, Citrix of Epic, whatever your limit limitation is when your particular network-positive, where you are were there services that you've deployed to deliver clinical care and all these different components that they themselves. So if you were able to ethic in a web interface and if that browser was a script, which it would be if it sort of the, for browser and the standards are allow browser reviews and were versions of things were taking monitored that would probably be better more defensible security posture than the one off print and science systems that some of the places.

[Man] The other thing too is that your particular system goes down but the independent for example. So like face interface, even if your endpoints or individual computers, workstations get Ransomed as long as you can bring in additional endpoints, additional workstations, you could still access the clinical systems. Without having to completely rebuild on other systems. So I'm excited to hear about an option, but em, again, attention to the details. And one other thing they would say is similar to your question about consolidation services into one paying potential reward, spread exploitation. If you make it faster base and there ends up being a vulnerability Chrome, for example, that's particularly nasty. Then everyone, would be horrible. Those things tend to get solved and patched much quicker than some of these other smaller systems because they're so widely used. But then again the devils, in the details. What are you gonna require? What standards are you gonna require one level, how scrutiny are you gonna put around, how secure those are gonna be.

[Man] I was wondering if you a survey to find out what browser is using it. Surprising how many people are using their really old one ID whatever single digit number that sign supported it doesn't have all those lenses. So our, our intent is mandate that as a system to do that cross board now.

[Man] Yeah, I had a quick question for you, whether or not one of the things that I've dealt with in my aggregation of data from multiple EHR all over the place. While I'm referring in my database. One of the things that has concerned me is that while, I consider all that data clean is electronic interface. The data's coming in clean, right? Have we seen yet anybody trying to attempt sequel, insertions, things of that nature into the actual medical record whereas these electronic systems, you know trust one another. They, they feel like it's all coming out or may not put the same network or, or or programming security into electronic interface because we trust the other system, they're not going to send over sample insertions or things of that nature. It seems to me like that'd be a great way to break into a system these days is they shall second transactions or move 'em all across. And as long as this patient can get moved around this patient gets moved around everywhere. Well, so it goes to piece of post snippet. It buried into the notes section. Anyway, the sun I didn't know if that's ever come across yet or.

[Chris] I haven't heard that as an issue at this point is certainly possible Christian perspective on that level. Although part of the you're just taking allergies, you're corrupting some clinical data. But I don't know if that it's a vector in necessarily the same way. We'll talk about that during the integrity section of my talk, but that, that question of can you modify data and how, what is your certainty that it hasn't been altered right now in EHR generally and most of the medical technology those assurances don't exist. We're a very trusting culture of data and a share, share, share, and we don't employ simple things like hashing to ensure data has not been manipulated. Your question about lateral spread and decode injection is a fascinating one. We can talk a little bit more but I haven't heard of that being deployed in the long end.

[Man] Seems like we've suspended the airplane ready for Christian to take it off this point. So I will hand off from here. I appreciate the attention.

[Moderator] Sorry, I just want to thank Chris for talking on a topic to meet the needs of our section meeting. You know, we chatted about it and he was up to the task of talking about a topic that, you know, he he put together this presentation for us. So we really do appreciate it, Chris. Thank you. So I am, I have the pleasure of being able to introduce Christian who I'm gonna pull my notes up. I hope I got everything right in your, in your stuff here. I apologize if I don't correct me., please. So, uh.