Session 1

All right. Hello people. And welcome. Hello. Hey. So, uh, Andrew, really excited. We'll do some, we have, um, Chris and Wes and Andrew here today. I'm gonna do formal introductions in a minute, but maybe Andrew, maybe just talk a little bit, little bit about, uh, the platform, um, and how this is gonna work because, uh, my community, um, we're, uh, may be new to the platform. Yeah. So we'll talk about that. We'll get started. We have a little opening, and then we'll get right into the exercise.

Yeah. Well, first off, welcome everybody, and thank you so much for coming. If you, uh, know anything about, uh, this collective group, if you've ever sat on the cyber call, if you've ever heard me and Gary, uh, interact, um, there will be some banter. Uh, it's all lighthearted and good in nature. Um, Gary talks about the platform he'd like to shoot this platform. Um, but here's the thing. With Crowdcast, there's some great things about it, but there are some limitations.

One of the limitations to Crowdcast is you're seeing our little Brady Bunch window here of four of us, and that is the limitation in terms of even being able to share a PowerPoint. So, for example, when Gary starts momentarily setting the stage and the agenda, and talking to Chris a little bit about what's going on in the industry, we're gonna move Wes off into the audience for, uh, a few, um, few moments. And, um, and then we'll bring him back.

So, uh, acoustic Echo, uh, Lawrence, hey, that's a, that's a feature set. Um, so, uh, but, uh, anyway, is, is audio overall good for everybody out there? Just a quick yes. From a few of you. Could you let us know? Is everything good right now? Yes. No, No complaints. No complaints. Okay. I'm gonna assume, I'm gonna assume everything's no echo here. Okay, great. Thanks, Dennis. Good to see you. Thank you guys. Reverb feature, not a bug. David Powell, welcome.

Okay, so with that, Gary, Here's all the yeses. We're good. Alright. Oh, a little behind. Yeah. Alright. Hey, we can get one or two more yeses though, just to be sure. So we got a great group. Um, we're coming up on like 200 and some odd here, um, alive and, uh, they keep coming on in. So, uh, alright. So let's do this. Gary, do you wanna kick things off? Is that good in terms of setting the stage with the technology? Yeah. Okay. Yeah.

So I'll, uh, can show my screen and, uh, who we're gonna move, who Over And Okay. We see in a minute. Okay. So let me, uh, all right. How we looking? You Look good. You're perfect. It's perfect. Okay. Alright. So here, here's what I want to do. Um, I, I only wanna take a few minutes to go through this, kind of set the stage for things, and then I'm gonna get out of the way and let this, uh, exercise, uh, happen. So, Andrew, you know, I, um, you know, we've been 10 weeks now on the cyber call.

Um, it's been so valuable. So we came up with this idea that said, look, I would like to bring this, uh, event to the True Methods community. Um, but also we said, if we're gonna do that, let's invite others. So, um, we've gotten great response for it. I think it's timely. Um, for, so those of you who aren't in our community, um, I'm Gary Pika, the president of True Methods. Uh, we are a company that provides, uh, training peer and, uh, software for VCIO and alignment to the MSP community.

Uh, Andrew Morgan is the founder, uh, ex true Methods employee, um, one of our first employees and, um, the founder of, uh, the Cyber Call. You're a founder of something now, Andrew. I know, I, I, I feel like I've broken out Gary, but, uh, appreciate all your mentoring over the years. Um, you know, it's, it's, it's great. I mean, you and I go, gosh, all the way back to the beginning days of ConnectWise. I think you were client number, what, 12 or something like that? Yeah, yeah.

So it's been many, many years, and I appreciate your friendship and, and mentoring. Awesome. Um, so, uh, for people, if you're not part of the cyber call, um, this is not a True Methods event. It's Andrew's event that I'm, that I'm part of, but we made an easy landing page for people to join. Go to true methods.com/cyber call. You can either attend Live on Mondays at one, or Andrew will send you the recording with a really nice WR written recap of everything that we do.

But even if you're driving around, listen to that, really keep you up to date on what's happening in security in the MSP industry. So, uh, you saw briefly Wes Spencer, he's the CISO of Perch Security, and on with us now is Chris Lore, the executive VP of So Security. And, you know, uh, these two guys are in this every day. And for Chris, you know, he's dealing with incident response, like, this is what his company does for a living. So there couldn't be two better people.

Um, I, I've seen them deliver this live and it's super powerful and they always keep it up to date with, you know, currently what's happening. Also, I'll bring this back up at the end, but Perch just released their 2020 security report. So basically arm yourself with the threat actor's perspective. So go to, uh, go dot perch security.com/sr two 20 dash tm. I, I guess, Andrew, are you teaching them how to have complicated, uh, places to go download things? Yeah.

Not in the simple addresses, but there you go. Wes, maybe you can put that in. Um, yeah, if you can put it up in chat so people can just link on it, I'll Take care of it. Sonny Lowe, great to see you, and congrats on your, uh, recent post there on LinkedIn. Uh, he's got an awesome company. Thanks. We, I appreciate It. He's, he's awesome.

So, yeah, I was gonna tell Wes, I'm like, they must have, um, been sent this, uh, link from the sales prevention team, and we have a great ebook you can do that has a SI cybersecurity checklist to evaluate your situation. True methods.com/security. That's how it's done.

So, look, COVID, uh, we, we've been seeing this increase in threats to MSPs, uh, really start to increase over the past 12 to 18 months, but over the past three to four months, it really has kicked into another gear because of, of COVID-19, the quick transition, you know, work from home, return to work, all all of these changes in terms of the topology of every SMB, right? They've changed in some way and imposters bad guys, threat actors, they have mobilized really quickly.

And one of the things is, you know, we're dealing against an enemy here that for the most part today, um, these organizations are larger, more mature than most MSPs or their customers. So these are not people in basements. Um, these are organizations that are well-funded many times they're government funded in other places that are in high rise.

So, um, even down to addressing, and I'll talk about it in a minute with one of the things that Chris is gonna mention, they're even getting smart about their packaging and pricing. They're, they're evolving. Uh, they're ransom and we'll, and we'll talk about what that looks like. It's, it's very true. Yeah. It's impressive.

So, um, Chris, here's just some of the things that have popped up over the last, you know, in recent, you know, weeks on, on the, um, on the cyber call, uh, things like credentials based attacks, people, if you could, just real quickly, they're learning how to get around, uh, multifactor or even use some things like single sign-on as a vulnerability. Yeah, that's exactly true. So nothing's really stopping these guys. They see everything as a challenge.

And, and most of the time these guys are well educated and well trained in cybersecurity. So for them, it's just another day at the office for them to, to, to work around these. What we feel are, you know, a year ago were very effective controls, and today they, they're working their way around them. Yep. Um, everyone's read right about the RMM, uh, and, and VPN vulnerabilities. I mean, they're really quick, right? Like they, they react very quickly. You can almost watch those ramp up. Yeah.

You can watch them move so quickly that you can't believe that it's not an inside job. So, I mean, they're just, uh, they know the, they know the RMMs and the, and the VPNs, all the major VPNs very well. So they know how to navigate through them and around them very quickly. A lot of times, I, I tell MSPs, they make the MS P look bad because they know their RMM better than they do A a. Absolutely.

Um, and, and if you think about it, like, you know, we mentioned on one of the calls, this is not going away. Um, you know, most of the vendors, not just rms, but most of the vendors that are successful in this industry, you know, they've been around a while, their code is 15 years old, so they have some technical debt, um, you know, so they, it's not easy to keep, you know, to make changes to your offering and keep it secure. So we'll continue to see that.

And a big thing you always talk about is change management, right? You, uh, you, you gave an example of you, uh, some RDP exception gets made for something that one person is working on, and there's no change management process, and it gets left open and gets exploited, Especially work, work from home situations. I mean, e every one of them seems to have today some, you know, well, so and so is too complicated for them to VP and N.

So we set up R-D-R-D-P just for her, uh, or, you know, we, we had one case that was pretty substantial, where it was an actual engineer in the MSP for some reason that opened up RDP to their RMM box. And so, I mean, the rest is history. I mean, he made those, uh, he made that attack easier than it should have been. Yeah, absolutely. Um, and of the cases you work on with SMPs, right? 90%, there's some involvement with an MSP, correct? Yeah. I mean, it's, it's, it's actually kind of weird.

If you get one that doesn't have some MSP in some capacity, it may be just full service MSP, which we typically see, uh, maybe a co-managed thing, or, you know, we, we've had that one case where the MSP was just there to provide antivirus software, and because they had their RMM agents installed in that, in that company's environment, that company got hit. That company didn't even know what an MSP was. They just knew it was the company that they bought their antivirus software from. So, yeah.

Uh, some way, somehow there's an MSP involved. Yeah. And, uh, I, I don't wanna steal your thunder, but you know, the last few weeks you really, your buzzword has been, uh, you know, um, the ex filtration data, ex filtration, and that, that's really where the advancements are coming from, of not only locking up, but having data they can ransom in different ways. And a lot of times you're finding the, the backups got deleted, right? They're learning and they're more patient. Yeah.

They're learning, they're more patient. They, when on the exfiltration side, they're finding the data that counts, that matters. Uh, it may be, you know, the obvious stuff, financial information, employee information, that type of thing. Or maybe something that could be embarrassing to, uh, the people involved, the owners, the principals, yeah. The executives, that type of thing. And, and these range and the ransoms range from 150 to 5 million. So it's, Yeah, for what we see, right?

I mean, there's some bigger ones out there. I mean, I've, I've seen and been indirectly involved in a $20 million one, but, uh, for us, you know, that's where we get up to that We pretty much max out the 5 million mark. Okay. So let me just take a couple minutes now that I, I want to kind of set for both, um, the True Methods community. I, I just wanna kind of unpack this a little bit.

And if you're not part of the, uh, true Methods community, um, it'll really set the stage for how I want you to look and why we have some of these challenges. So think about this, A company with a thousand employees, um, probably has three to five resources, uh, or more depending upon their industry. But three to five security resource resources on staff, how many dedicated resources, even just proactive, not even security, right?

Does an MSP with a thousand seats under management have, um, most of the ones that I deal with, especially outside our community, right? We teach proactive roles. Uh, you can round down, uh, Andrew to zero for most of them.

And when you think about that, and when you think about, um, what you're gonna learn today through this exercise of how much work and process and discipline in addition to tools it takes secure customers, uh, you'll understand why right now, um, it's not really a fair fight between, uh, threat actors and MSPs and their customers. So this is our true methods framework.

This is what we've been teaching for the past 10 years, to separate what everyone does, which is to have a tool stack or centralized services to provide support. You know, you do projects and give some advice to really defining truly a hundred percent dedicated, proactive roles. And this idea of a TAM or technology alignment manager where your documentation, your standards, your alignments, where this is where compliance happen, right?

Um, and then being able to translate that through the VCIO to customers. This is where you need to have dedicated, proactive roles if you and your customers are going to be secure moving forward right now. And so this is a critical, critical piece of it, and it's the reason why we are where we are, because an IT director only has to go to the board in order to ask for more budget and explain what their risks are to get another resource.

But if you're an MSP, you need to figure out what to do, operationalize security for yourself and across all your customers, then you gotta figure out how to package and price it into your service offering, then go to every customer and every new customer and command a higher value so that you can afford to fund these additional proactive resources. And, you know, Andrew MSPs are good at a lot of things, but this part here, for the most part, not so much.

And that's why we came up with this concept, right, of omics, which is, hey, we have these roles, we know which ones are proactive, and we're gonna come up and we're gonna determine what the cost per seat is, so that we know, just like the IT department is gonna fund roles that are gonna reduce their risk, we're gonna also fund the roles of the process and the tools.

So we use this, um, we call this the, the, the seat price or seat cost, uh, calculator, where you can go by roll, put the labor and under centralized services tools to be able to determine what your average cost per seat is. So you know, what, what price you have to sell at that you'll be able to fund these roles and process, because without that, your intentions around security can be good. But if everybody is billing an hour or doing a ticket or alert, uh, it's not gonna happen.

This happens every day, happens every week becomes a culture, but you gotta be able to afford to have that culture. So does, does that kind of real quick? I don't, I'm trying to get right to the event, but that kind of sets the stage, right, Andrew? Uh, absolutely. Gary, I was, I was keeping myself on mute, but yes, absolutely it does. Okay, awesome. So then here's what we're gonna do, Andrew, I'm gonna let you move me off.

I'll be, uh, chatting, uh, on the side here, um, through the event, but I'll let you move, bring Wes on, and, um, and, and, uh, I'm gonna, I'm gonna let them do their thing. Yeah. Stay. Just give, give us one minute, Gary. I just stay, stay here. I'm gonna pull Wes up and I threw a, a quick poll out, and I think this'll kick things off. Um, it's just really telling, and I'm just gonna wait for Wes. Uh, you should be here any second as I'm pulling them back up. Hey. Hey, Wes. Hey, hey.

So I did a quick poll, um, and Richard, I just said, Hey, have you dealt with an incident since Covid began, either internally or, or with your customer? By the way, I, as, as, as I said, the, the, the polling system here does not collect any data, so it's just a yes or no. So, Gary, it, it's basically 4 60, 40, no, 60%, yes. 40%. Like, I mean, it's, that's outrageously high percentage, right? Yeah. Well, that's my point.

You know, I just, you know, maybe Wes, you could, you know, get your thoughts and, and Chris, and then we'll kick it right to you guys. But basically, I mean, it's almost a flip of a coin at this point that your customer is going to get a, get hit, get popped Some point. Yeah, no question. Right? So, um, Wes, you guys seal a lot of stuff right from your sock. So Y yeah. And a few things about that. I mean, it, I saw this in the chat, it depends on how you define an incident. Sure.

It also de depends on if I were answering that it'd be yes or not that I know of, right? Anytime my board would ask me at the bank, Hey, are we doing good? Are we secure? My answer is always, lemme go check again. I'm not sure right? Because things change. Um, but yeah, I mean, even seeing that 40% as an affirmative, yes, that data is telling, right? I mean, we know for sure without any doubt, uh, there's been a pickup of attacks, there's been a focus on it.

I remember at the very beginnings of the pandemic and the initial shutdown in work from home, we saw RDP attacks just skyrocket. They've come back down now. Um, but there's no question that, uh, the attack surface has changed and bad guys are aware. Yeah, yeah, yeah. Chris, surprising you all to you? No, not surprising. I mean, I mean, you have, you know, the ransomwares and the business email compromises are, you know, the, the main two.

And I would say, you know, from a pure volume perspective, the business email compromise is number one, they may not take as long or, or or to, to, to handle as a ransomware case, but, uh, they are a plenty and they are very, um, very damaging financially to, to, to most companies one way or another. I mean, especially reputationally.

I mean, it really sucks when a company has to send out a communication that says, Hey, uh, you may have received an email from us earlier last week, that email did not come from us, or blah, blah, blah, blah, blah. And so, especially if you're a company that's an accounting firm, financial services or something like that, where, you know, you're kind of regarded as one not to do that and you do it, it's, uh, it's pretty embarrassing. So now this doesn't surprise me at all. Okay.

So, hey, so just in closing, I'll let you talk in a second, Gary questions. Bring them in. So there's a, there's an area here. It's, I know it's easy to go to write, to chat. Can I ask everybody to put the questions in the, there's a ask a question section. There's already been one asked. Um, I'll save them throughout the, the, uh, you know, event when Chris takes some breaks and Wes takes some breaks.

But, um, it's, it's just like, if I, like I said, if I could ask that you guys do it there, because it's hard to keep track of all the banter going back and forth with chat. Gary, you were gonna say something before we turn off? No, just Andrew.

Like, um, if everyone stays to the end, if you bring me back up, you know, five or 10 minutes, whatever you got at the end, and I'll just, uh, I'll be on the side chatting with people and also, uh, you know, kind of making some notes so I can pull this back right to, okay, you learned all this stuff. Now what do we do with it? How do we move forward? How do we take this and actually apply it in an MSP business? Sure.

'cause without applying it and understanding, and it's gonna be a process and start to set goals every quarter of how you move forward and give people a little bit how I, in my training program and in my peer group, wh where we're taking people to help him do that. Yeah. And Chris, Chris, let me ask you, will there be maybe a few breaks, maybe every 20 minutes or so we could bring Gary back in and, and kind of talk about where we're at so far and kind? Would that make some sense?

Yeah, I mean, we can do that. I packed a bunch of stuff Okay. Today because I, Yeah. And I don't wanna get in the way I packed a bunch of stuff. Okay. Yeah. You don't worry about me, you just bring the value, pack it in, and I, I'm fine. So, uh, don't worry about me, I'm good to go. Alright. Alright, Gary, we'll see you in a little bit. Awesome. Yeah. As, as Gary transitions off and we bring the PowerPoint up, I do want to just thank Gary and the entire True Methods team.

We would not be having this today and would not have a blowout of almost 600 people on this if it were not for Gary and what he's doing. So true methods is awesome. And Gary, thank you so much for bringing us on board. Um, Jack, I see how long this call will take. That's a question, um, that just came in, you know, we'll see. It's probably gonna be a little over, what'd you say, Chris? Maybe an hour and a half or so. And keep in mind, this is recorded. You can always come back to it.

Uh, and you can also hit the pause button, by the way. It'll go off live and you can come back to it. If you have to take a bio break or something, um, just know that you, you'll be behind us, right? So if you're asking a question and you're watching it late, we're gonna not know what you're talking about. So just know those things. Yeah. Can everybody see, you know, uh, Eric said I lost the stream, I'm assuming. No, I'm not seeing, yeah, We're good. I think we, we've got it. Okay. Well, good.

Wes, Chris, I'm gonna, um, turn off my video. Um, I'll be here with you guys in the background, but I'm gonna let you take it from here. All right. Everybody sees the presentation. Yeah. Uh, it's perfect, Chris. All right, fantastic. So I appreciate everyone getting on. Uh, for those of you that have been involved in Wes and my tabletop exercises, thanks for coming back.

Uh, we try to do, uh, the best we can here virtually, uh, we know we've done a number of 'em in the past in person, and those are really exciting and really fun. But we hope to gain as much out of this. I hope you gain as much outta this one as you do those. Uh, I just, like I'd mentioned just a, a minute or so ago, I have packed this slide deck with a ton of information. And so, uh, please be aware that we do have about 90 minutes. We do want to take some breaks to get some questions answered.

Uh, give Gary some time at the end to kind of wrap things up in Gary fashion. Uh, but I have gotten a lot of feedback from you guys and you really wanting some, some help with solutions and stuff. So I packed just a bunch of data in there. Um, we're only gonna do a single scenario today. Uh, this scenario takes place over weeks. So obviously I've condensed it down. It's very accelerated. Uh, we will have recommendations in here.

Uh, we do want as many people in your organization that are responsible for incident response involved in this. Uh, we will be, provide answers and advice. There is a lot of content in these slides. And I'm, I, you know, I think we, we, we could spend a day, uh, doing this, and I think Wes would agree, we could, uh, but we're gonna do a lot of this.

And, and again, it is very fresh in what we're seeing out there, not only from, uh, this scenario, but also from what we see people struggling with, uh, especially MSPs when we get involved in these cases, what they're kind of struggling with. And so, uh, a lot of the recommendations that advice are current to that. Um, we will be making a, a portion of these, mainly the recommendations and advice portion available to you. I don't know whether or not I'm gonna release this scenario yet.

Um, because they do take some time to come up with. And, um, you know, uh, I'm not a Hollywood screenwriter, but let's jump in. It's July 5th, it's after the long weekend. Everybody is returning, including your customers. You started your morning huddle, and with all your employees working from home, your dispatcher jumps in panicking. Uh, phones are lighting up, emails are coming in, customers are experiencing issues, accessing their work, their networks. And there is no consistent story.

Customers are just saying different things in their own kind of vernacular, in their own, in their own way of doing things and saying things. Your lead engineer, he's off fishing and he is completely off the grid. Another engineer chimes in and says, the access to your arm m is now not working. That the server is up, but the application is screwed. Your PSA is also affected, but no one knows why. You as an MSP, as a, as a person in charge here are a purist.

And you do everything on premises with a small exception of email. You do have Microsoft 365 for your email, but that was a big giant step because you've been doing exchange for like 25 years and you love it to death. And that was pretty tough. But everything's on premises. Obviously. All of your employees are online today, except for that lead engineer. They all know something bad has happened, your phones are lighting up. Your RMM is down, and you manage over 10,000 endpoints.

So think about it for about a minute. You know, at this point, knowing what information you do, you're getting a bunch thrown at you at one time. In this particular scenario, what are you gonna do as part of your incident response plan? And it's too early to start drinking. So that is not an option In this particular case. Plus you're hungover from the weekend. All right. Uh, I got some questions for you after you've thought about that for a second.

Are you as an MSP, as whatever position you you're in, are you prepared for this situation? Do you have an IR plan that is readily available? Uh, do you know what to tell your staff at this point? This is a different situation than, uh, none of your staff may be in the dark. So telling them something a little bit more generic, uh, to kind of keep the things.

'cause a lot of times we like to preach that you want to keep, uh, what's going on as far as specifics, very close right now, because you don't want anybody to say something in error. You don't want somebody to, uh, leak something to a spouse, to a coworker, former coworker, whatever the case may be. Can you function without your lead engineer? And this is a big question. I mean, we've been in a number of situations where that lead engineer, they are truly almost helpless.

Without that person, do you have any preliminary messaging already set up for your customers? And do you have alternative ways of communicating with your customers? So if your RMM is down, your PSA is down, um, you know, you may not have access to that type of information right away. So do you have ways of communicating them outside of those two major tools in order to do that? And Chris, a couple things I wanna mention on this too. Sure.

And I think the pandemic has really taught us this is, um, cross training, right? And I want to ask this question into the chat. Um, can you give us a yes or no? Does your MSP do cross training? Um, I'm curious to get a yes or a no. So the way I define cross training is, do you have multiple employees that all can do, uh, different roles? It may not be their day to day, but they are capable of filling in the gap. Um, Jeff says, we're on it. Alexander. Yes, Paul. Yes.

So I'm seeing a lot of yeses and some sums come in. Yep. And it's okay to have a no as well, right? The goal of what we're doing today is the purpose is not to, this is not a pass or fail. A tabletop test is never about that. A tabletop test is about questions we ask ourselves to see what we're doing, what we're not doing, and where those gaps may exist or where we could even mature further.

So, glad to see some yeses do see some, some sums coming across, but cross training is vitally important. Yeah, thanks, Wes. I mean, it is, and and we're not just talking about cross training technically right in, in an instant response situation, you need to, you may need to cross train on incident response roles. You may have somebody assigned to dealing with, uh, the public relations side.

And if you only have one person and that person's out for whatever reason, uh, you need to have somebody cross-train there. So don't think about it as just an everyday type of cross-training. Think about it as cross-training, and under the guise of incident response, here's some advice. You need to locate your plan, and you need to dust off that thin layer of dust that's on it.

Um, I was in a conversation with our director of instant response this morning on the forensic side, and one of the things that he mentioned is that we, we, that, you know, the instant response plans just overall are just not updated enough and are not refreshed enough.

And so it's a living document, and I think it's gonna be one of those things that if you're not handling it as so, and if you're not reviewing it on an ongoing basis, it should never have a thin layer of dust, whether that's a actual layer of dust or a physical or virtual layer of dust. Excuse me. You need to determine who on this team needs to be involved directly in this case. And you need to d determine who needs to be involved indirectly.

And this is very key because you're gonna have two lines of communication. You're gonna have communication that's with your very, um, close, uh, incident response team because there's gonna be a lot of decision making, a lot of information and stuff that needs to stay within a very controlled set of people. And then you're gonna have a different stream of management and communication and others to the others. So you, you need to know who those people are right away.

Typically, they're defined in your incident response plan, but if for some reason they're not there, whatever the case may be, you need to figure out who that is. Then, um, you need to communicate for people what they're gonna do and what they're not going to do. And this is really key, and this is what we're talking about, is you need to tell them, Hey, you are gonna communicate with the customers, or you're not going to communicate with the customers.

You are gonna say these words and use these words, or you're not going to use, and you're not gonna use these words. You're not gonna use words like hack, breach, compromise, and so on and so forth. So in your plan, you really should have some language already put together, and maybe it's four or five different options of language that at least you can choose from that people are familiar with. And you're gonna go, Hey, we're gonna go with option A on this one.

You need to call the experts early. Uh, especially in a case like this where it's so freaking obvious that, uh, this is, this is a major deal. You don't need to be, um, trying to just do your own thing here. And typically at this point, especially technical people, we're not thinking about preservation.

But preservation is so important and needs to be just in bold and, and whatever ways you can put it in your incident response plan, because I'm gonna tell you right now, and this, I'm gonna echo this through this conversation in a, in a case where the MSP is the root cause or as part of the root cause, whatever you wanna say, that forensic information is so important to you, and it's so important to your customers.

Without that forensic information coming from your systems, it's gonna put everybody in a, uh, in a bad mood. And it's gonna make for a, a, a much longer battle, uh, than you ever thought would be. So, and then also, I think this is where you just don't start pulling triggers, pulling levers, doing that type of thing. Uh, you just need to, you need to take some breaths during this time. Uh, you're gonna feel pressured to put your foot on the gas.

You're gonna feel pressured, uh, to just jump the gun and start doing things. And your staff is gonna be doing the same thing. So here's where you need to be very well prepared, well rehearsed, and, uh, and, and make sure people are doing things very intentionally. So Chris, I wanna, um, talk a little bit about the value of playbooks in all of this, and we're gonna come back to playbooks often inside of this, uh, tabletop today.

But a question for the chat that I'd love to get back, do you guys have playbooks that includes standardized responses? So for example, um, you know, let's say it's a, like a malware incident of some kind or some kind of wire fraud or whatever it may be. Do your playbooks include exactly how you would message that? And then you might just put it a few words at the beginning or end so that when you have an incident, you immediately know the framework of what to say?

Yeah, lots of nos coming back, and I thought it might be that way. And so, you know, one of the things that's really good about playbooks is you don't have to use it prescription based. Like you could potentially have it say, you know, give you an outline of the things that you would do. And I'm gonna make another sports analogy.

I've been using sports analogies a lot lately for some reason, you know, when, uh, football teams go in and they get ready to play, one of the things they often do, and college teams do this a lot, they actually, they have a playbook, right? And they actually prescribe that playbook. And maybe the first 10 plays of the game, their offense is running prescribed no matter what. This is exactly what we're going to do because it lets you hit the ground running.

It lets you really get involved in the game and not make as many mistakes. And so playbooks are really, really helpful to document how we would do different things, whether that's a response, whether that's how I write something and, you know, on the PR side, whatever. So definitely, definitely give playbooks some, some love because they need it. All right. We will be referencing IR plans and playbooks throughout this. So here's the next scene. A couple hours have passed.

You still not called anyone for help. Your customers seem to be patient and are accepting of your explanations. Your nephew, who you had to hire has taken it upon himself to call RMM and PSA support for their help. You still do not know how it happened or when it happened. You had MFA enabled. So you are, you are baffled, your customers are holding patiently, it seems. You still don't know what'll get exactly has happened.

And it, you are, it appears right now, you are hesitating to call for help at this point. So think about yourself in this situation. And you might say, Hey, heck no, I would've called insurance right away, but think about this in this particular scenario and tell me, uh, you don't have to tell me, but think about your next steps. I'll give you a few seconds here to think about this and discuss it with your teammates if they're on the call with you. Yeah, and that's the goal, right?

I want to come back to that. If, if Chris and I were doing this live, we would have everyone at round tables and we'd wait, we'd be listening to everyone discuss, we'd come back and you ask questions. We're not, we can't do that right now because of the pandemic, right? But, you know, definitely pause this thing if you feel like you have to, or you can watch it through, but come back and pause at these events because what you really need to do to make a solid tabletop, you gotta discuss.

And so our job with what Chris and I can do is we can give you things to think about, but unless you and your teams actually sit down and discuss it, uh, you, you won't really get the value out of this. So, so that's kind of the, the big goal here. Make sure you guys afford yourselves time to do that. Yep. You gotta poke holes in this thing.

You are a very complicated business entity, so you are, you're almost like a multinational corporation with hundreds of different and diverse businesses that you're responsible for. So keep that in mind. Yeah. And, and Jason, this is recorded. So I question in there so we do have it all paused or the ability to pause. Alright, fantastic. Don't, so here's my advice on this one. Don't go cowboy or cowgirl, please. Um, just don't do it.

I mean, uh, I can't, I can't re I can't tell you how many times that because it's an MSP and because it's a technology based company and because so many people there have so many years of combined experience that they can handle this. They can take this on their own. They went to a webinar, they're trained, or they have their C-I-S-S-P or whatever the case may be. You don't do this stuff every day of your life.

And so there's really no reason for you to attempt to do this every day of your life. Uh, you need to be thinking about attorney-client privilege. And this is a big deal these days. Uh, there was a recent case with Capital One that has a lot of the attorneys kind of buzzing around. Uh, capital One had a, an existing relationship with FireEye for incident response. They got FireEye involved first, obviously because they had a relationship with them. They got their attorney involved.

Second, when it went to court and there was a, somebody wanted a discovery motion on the stuff that FireEye had done. Um, the client had argued, the defendant had argued that it was covered under attorney-client. The JO judge ruled otherwise because the relationship was a preexisting relationship and therefore shouldn't be covered. So that's somewhat of a unique situation. However, it does have, it does have the attorney spook from attorney-client privilege perspective.

So you gotta be thinking that in mind. So the earlier you can get that involved, the better off you're gonna be. Um, you need to continue the delivery of, and the key word here is consistent messaging to your customers. You don't want anybody opining, you don't want anybody tweaking the language. You don't want anybody, uh, caving into the pressure of a customer beating you up or whatever, beating the employee up or whatever the case may be.

Don't forget about the customers you've already communicated with as well. I know this is still early in, but this is advice that goes throughout the process, uh, throughout, throughout managing the event. Just because you've told the customer and they sounded like they acknowledged you and they were happy, doesn't mean a couple of hours from now they're gonna still be the same way. Uh, so don't, don't assume that they're satisfied, even though they checked in two hours ago and seemed that way.

Uh, get in front of the call to support. So this is the one that your nephew called in. It probably won't be that big of a deal. The problem though is, is it could be, I just think it's one of those things that people don't pay attention to, uh, because they divulge too much information, too. Support and support knows way too much, mu too much information. And so it's not that you're not gonna get support involved, but it's gonna be premature.

So you want to probably escalate, uh, if that call was to happen, you want to escalate to somebody that has a fairly senior role there to make sure that things are suppressed correctly there. And then you do wanna try to have somebody start documenting this stuff on paper, if possible.

Uh, keeping notes, uh, of what's going on, you know, conversations, uh, what whatever At a high level, not, we're not writing a book here, but what we're trying to do is just keep some written itinerary of what's going on, uh, kind of internal meetings that have taken place, those types of things, and do it on paper for right now. Uh, and don't, don't send things through email if you can, if you can avoid that.

Hey, Chris, a really good question just came in on the chat, um, which was, uh, you know, can you give an example of saying too much? And I'm, I want you to answer that, but the one thing I wanna say right before that is, this is a problem that I think we have in MSPs because MSPs are so service delivery oriented. I mean, it's our nature to want to help. It's our nature to want to give back as much as we can say, here's what we know and here's what we're doing.

Especially if someone's really angry and yelling or very concerned, right? Um, but it can be dangerous to say more than we should. Um, can you speak about, like an example of saying too much this people have a frame of reference here. I know we're not lawyers, but helpful. I, I'll I'll give you some. Just, I mean, typically we just hear people say, Hey, you've been breached, which is not by definition actually what has happened.

I mean, it may be technically what did happen, but you're not in the capacity to make that type of statement. Um, you know, or, um, you know, it's a ransomware attack or, you know, some hackers have, have hit us and we're down as a result of that. I mean, those are the types of things we hear. I mean, the other things that we've seen lately too is we've seen a, uh, a company send emails pretty early on with the word findings in it.

You know, Hey, we found that, um, re remote desktop was open and an attacker, it appears that attacker has been logging onto your network for, you know, six to eight weeks. That is very premature at that point. And, um, you know, again, it comes back to this attorney-client privilege thing. And so that third party involved that was giving that assessment, um, or giving those findings, whatever you wanna call them, was not engaged from an instant response perspective.

And so that it is possible that that could be discoverable. Again, I'm not an attorney, we're not attorneys, but that's how I have been told to understand that type of situation. So the next scene you've called insurance and they have assigned you a law firm and an IR firm. The IR firm starts to get dug in and analyzes what they can do. They find a ransom note and identify the variant as soden. The IR firm tells you that these guys are known to exfiltrate data.

They visit the Soden TOR site and discover the demand is $5 million. You're also confident your customers are all hit to what degree is impossible to tell. Your backups are still, your backups are still under review and your client's backups are safe due to you doing them the right way. So, um, your backups for your internal backups, um, for some reason that wasn't on here, but your own internal backups. You don't know the state of them.

You've done a great job with your customer's backups in monitoring them, but you have not done a great job with your own backyard, with your own backups and so on and so forth. So you're not even sure at this point whether or not you can even recover anything from backup for yourself. This is a sample soda note. Um, so just you can kind of see what it is. Uh, this is basically what you would see when you could go to the website. Some of the stuff I've kind of redacted out.

But you can see here that this was one that was a $300,000, uh, uh, so in demand. And the other key to see here is they are not asking for stuff in Bitcoin. They're asking for things in in Monero instead. And so the challenge, they do that because, uh, they believe that, uh, doing it in Bitcoin has a higher chance of traceability.

And so Monero, uh, does not have those characteristics as well as, um, but I will tell you that, uh, making a payment in Monero takes quite a long time compared to doing Bitcoin. And I don't know, Wes, if you wanted to add anything from a Yeah. Perspective. A few things I wanna add. Yeah, this, this brings up so many things. Uh, maybe we'll cover a few of the common questions, Chris, that we can asked around all of this.

So first of all, generally speaking, it's a very poor idea for you to hold Bitcoin. We get asked that question all the time. You know, Hey, what if I hold X amount of Bitcoin, you know, myself? Um, that's a poor idea. Um, only because not only do you wanna have to hold, maintain, who knows what the, the ransom is going to be. There are quicker means and mechanisms of if you have to pay the ransom of acquiring that, and you should not be the one doing that.

There are legal ramifications around all of this. Um, in fact, you're seeing this in even state regulations, for example, what, uh, Louisiana's doing, requiring MSPs, um, to be registered and also for client or for, for government organizations inside of Louisiana to to register. They have an Ms P because they need to know the paper trail and monies move movement, especially to criminals must be done. I hate to say it, but there's a right and wrong way to do it.

Uh, and generally speaking, you should not be the one to do it. Hey, can we get a woowoo? We just got 600 people on the call. All right? Yeah, yeah, yeah. That's awesome. So Gary Pika is going, Gary Pika Jr. Is gonna do a back flip. He promised us after the call, um, okay, so back, back from your, uh, regularly scheduled message. So, so don't, I would not recommend that.

The other thing too, and I just wanna mention this because I'm a crypto nerd, is you, you do see, um, things other than Bitcoin because there is hope of recovering the actual Bitcoin. Um, what will happen is after the ransoms are paid, uh, and what will, uh, sometimes occur is insurance companies will monitor those wallets and, uh, bad guys do try to wash and cycle them, but in some cases they move some of those known malicious funds over to an exchange.

And if the US has, if it's a US based attack, if if the US has extradition authority over that exchange in another country, for example, Europe, they can seize that Bitcoin. Um, so there are chances of getting it back. Uh, I just wanted to mention that, but don't have a hope in that, right? The goal is to never have to pay a ransom. And of course we'll get into what happens after the ransom later on this too. So Chris, I just thought I'd focus some of those things in. That's, that's good stuff.

So what are you gonna do at this point? Are you gonna cooperate with insurance and legal? Uh, to what extent are you gonna cooperate? What are you gonna do now to direct your staff what to do? What are you gonna tell your customers of anything different? Do you still want your, your customer support involved on the RMM and PSA sides? Have you thought about standing up a new instance of your RMM on your own? Oh, so think about those things for a minute or partial minute.

What is your confidence in your backups at this point? If, if, if it's been hours and you're still not getting a firm answer on your backups, um, what, what's your gut feel that they're gonna be good? What steps can you take to get more clarity around your customer systems? You, you have a feel, you, you think you're pretty sure that they all got popped, but you don't know if, excuse me, to what degree are you ready to turn the keys over to the IR company?

We find this is a really important question because we see people that just want to interfere. We had a case late last year with an MSP who got hit, and um, they have two principals in that particular company, and it's a husband and wife team. And, and one of them just did everything they, they could do to roadblock us every step of the way to help them. And so, uh, that happens and it's happened more than once.

And so we've had another example where the, uh, the systems, both of the customers in the MSP systems were, uh, encrypted and we were ready to decrypt them. And, uh, one of the principles of the MSP demanded that we take care of his home system first before anybody else's. So it's, it's interesting how these things go down. What is your emotional state now?

So assuming that you're a principal or you're, you're, you're an executive or, or you're the, the owner of this, this process, what's your emotional state at this point? And this is a very personal reflective question that I want each and every one of you to think about at this point. Um, are you mentally and emotionally capable of handling situations like this? This is a different type of firefight than your, your customer servers going down or a firewall going down or something like that.

Uh, this is a real hit in the gut at the same time. And so the reason why you asked this question of yourself is to make sure if you're paint playing a key role in an incident response process, you need to be willing and able to do that. And if you can't, uh, then, then you need to get out.

I mean, again, I mean, you would think these were more exceptions, but we've actually had, we had one situation where the principal of an MSP actually left to go to 'cause 'cause he had a appointment with rehab and he was off to to, to a short rehab facility right in the middle of the darn thing. So, you know, you need to answer those questions and make sure that if you have that role defined and instant response that you're able to carry it out.

Are there any customers who are you are who you are really worried about? And are you or someone else going to work with the attorneys? And not to steal Wes' analogy, but think about incident, think about the sports and the football analogy where you have third down backs. And so that, you know, there's guys that sign on to teams and make a pretty good pinch of money, uh, just for the standpoint as they are specialized in in in third down place.

Uh, same thing kind of goes with special teams, right? So in an incident response situation, you, those roles may be changed to someone that is more like a third down back. That's someone that's better equipped to handle an incident response situation, both technically and, uh, any emotionally. You need to embrace IR and legal, I can't, I can't overstate this. Uh, they are definitely there to help you and they are to engage you with you directly.

One of the questions we get often is, well, you work for the insurance company. How do we know that you're not doing things in the best interest of the insurance company and not me? That's just not the way the thing is designed. The thing is designed for, for them to, to work with you and to get you back on your feet as quickly as possible, uh, and your and your clients. And that's in the best interest of you, and that's in the best interest of the insurance claim.

So there's not this, um, huge, um, kind of sadistic conspiracy to, for insurance companies to figure out ways not to pay out on your claim. Uh, you need to find a manual process of, of, you know, getting a tab on your customer situation, whether that's phone calls and asking those questions. You probably need to have somebody that's capable of being calm, cool, and collected on calls. But you need to do that. You need to, you need to have your customers.

If, if the only place that you're keeping customer information is in your PSA and your PSA is down, you got a real issue. Uh, so you need to have some type of offline repository of this information to where you can have something that's easily shareable, uh, with legal counsel and ultimately with, uh, crisis communication as well. Uh, you need, you need to have as much information on there that an attorney would want to know, like, you know, what industry are they in?

Maybe what percentage of revenues that they account for, for you, number of employees they have, what states they operate in, uh, that type of thing. 'cause that's gonna be very important. This whole issue around, uh, the different states and the way they are handling these types of things is, is, is ridiculously important. Now. It wasn't as much so a year, year and a half ago.

Uh, but the state laws are, are not only increasing, but also just the level of scrutiny, uh, by the attorneys general and so on and so forth. And so having that information ready. Now, if you have customers that operate outside of the us that's even more important, um, especially now. 'cause there's a lot of, 'cause internationally, many countries are ahead of the game from a privacy and security perspective than the, than the US is.

So sometimes you have to bring in different council to address those international concerns. Um, exfiltration, I've been kind of plugging this, if you guys have heard me on the, on the Monday calls and so on and so forth. Uh, this is a big deal. It used to be like with, a perfect example with this one was so into Kobe, we used to say, Hey, we're with these guys. They're, they don't exfiltrate data. Uh, we don't expect to see any exfiltration of data. And so that makes everybody feel, feel good.

We cannot say that any longer. And it's very rare that we use those words anymore. I mean, exfiltration is almost a given. It's almost the only time we don't see exfiltration. It's typically if it's a really small type of attack. Uh, but if it's an MSP attack, you can pretty much be certain that they're gonna exfiltrate something from someone. And so the lawyers are gonna wanna be informed about all the potential data aspects of that.

So you say, well, what, what they, this is where, know your customer is very important. If your customer has a bunch of SaaS stuff that's not on the domain, that's very important information because the likelihood of that SaaS stuff being accessed is probably gonna be pretty low. The attackers these days just aren't going to those, uh, steps to, to do that.

Uh, however, if you know, your customer has everything internally and they have, you know, a decade plus worth of files on a file system and or, you know, whatever the case may be, um, you know, if they're a personnel company and those types of things, those are the things that the attorneys are gonna wanna know off the bat so they can start preparing and doing the research on what needs to happen.

From a notification and reporting perspective, you need to have somebody assigned, and I'm hoping you would already have this ahead, but of, of doing the preservation. And I'm gonna tell you that firewall logs are typically not our first choice for doing forensics, but they are the, they are our, basically our backstop if we don't have the forensic data we need.

And so the firewall data about when people access, how long they accessed, uh, if we're able to, uh, to determine how much, uh, data went in and out during those sessions, uh, that can help a lot. Especially with exfiltration cases where we don't have the level of detail we need to tell what the attackers took and what they did not take. So, um, I can't tell you how many times that we, we've told people to get this done and they put it off.

We had a case recently where they said, Hey, we got, we looked, we logged into our firewall. This data's gonna be there for another 27 days. Uh, we won. We'll grab it, we'll grab it some other time. And then they forgot about it, even though we kept asking them and asking them to ask him about it. So, uh, it is crucial to collect as much of that forensic data as you can early on. The IR firm will give you direction on that, but knowing ahead of time that that's stuff.

Now, if you have a solution like perch in place and you're collecting those logs in perch, your job is done. But, um, if, if you don't, and you gotta do manual stuff for your customer and for your customer's sake, if they don't have their stuff in the perch, shame on you. But that needs to, that needs to be a, a high priority. Hey, Chris, let's talk about a few things here. So, a question came in from Jeff that I thought was a good one.

It said, is it possible to have an engagement letter with IR ahead of time proactively so that when an incident happens, we're good to go on that part? You wanna answer that? So I'd say so there are, so there are a number of people that are doing that today because they're finding that they may not want to always file a claim. So it may be in their, in their best interest to kick off an incident response process with the third party IR firm and even a legal firm before making that decision.

And that's your prerogative. But if you just have the IR firm engaged and not legal, even if you have general counsel, it's better than nothing because if you just have IR firm, you're gonna go back into that attorney client privilege risk situation again. But it is, I I, I, I'm starting to see a trend of people wanting to have that I firm at least to get them involved initially, to give them some guidance to be a part of that incident response process in that plan.

So I definitely think, um, having an IR company somewhat preen engaged, uh, is helpful. Yeah. And, and it can be expensive, right? But enterprise is certainly that way. They've preen, they've, uh, pre-negotiated and have on retainer I an IR firm, often, not always, but often. And one of those reasons is you mentioned it's a level of comfort of who they work with. That's not cheap oftentimes, if you go that route.

Um, uh, and, and this brings up a, a question, Andrew, I don't know if you're still on, but I wondered if you could create a, um, this really, Chris gets into this idea of cybersecurity insurance, of what level of comfort MSPs have around their cybersecurity insurance. Do they know, would it engage IR and legal for them? If so, you know, what would it be? What are the limits and coverages?

Could you just write a poll question in that just pulls, everyone says, what level of comfort do you have around, uh, or what level of understanding is probably a better way to say it, do you have around your cybersecurity insurance and just like, high, medium, low, something really simple. Okay.

Um, really curious to get everyone's feedback because you know, Chris, I don't know what you think, but I, I think we get a lot of questions and there's a lot of confusion and worry that, uh, MSPs have over their cyber insurance, and if it's adequate and if it covers the things they need, wouldn't you say? Yeah, I mean, it is a concern and, and it's, um, and, and understanding how your insurance plays a part and how your customer's insurance plays a part is very important.

Um, you know, it's, it's interesting because, you know, we talked to some MSPs and they think, well, all of our customers have cybersecurity insurance. So we're, we're in the clear, but it's not that easy because those policies, uh, claims could come back after you. So if you have a, a, an insurance policy in place, typically their insurance claims are gonna, you know, what's called subrogate against your insurance policy.

But for some reason, if you don't have a policy or you, uh, your policy is maxed out on its limit, uh, those could turn into to legal fees. So you need, um, like lawsuits. So you need to be very familiar with that and understand the, the limits, the sub limits, and understand the process of engaging with insurance and, and what you're allowed to do and what you're not allowed to do. So Chris, I'm looking at the poll data coming in, and this is exactly what I thought.

Um, most low, uh, medium, and then a few highs, right? And those may be the lawyers that are on the call, for example, right? But this illustrates what I thought was the case, and this may be another takeaway as well. Chris, you wanna give some advice around those that are around low. Uh, what they could do is that just require, uh, and would you recommend a call with their insurance company? Or how can they go from low to something higher in a case like this?

Well, With cyber policies, they're fairly broad in the way they're written. So they're not a very heavy duty, deeply, uh, legalized type of document. So if you actually take the time and actually read through it, I think for the most part you're gonna understand it. Uh, but you can discuss it with your broker would be the first step to take.

And, uh, your broker can either explain things to you pretty pretty quickly or chase it down from the actual insurance carrier, the on the back end for you. Um, there's also coverage counsel, those, those are attorneys that, um, are involved in determining what is covered and what's not. A lot of times people get coverage counsel engaged after an event if they feel that insurance hasn't, uh, responded correctly or, or insurance is not covering certain things.

Uh, so coverage counsel gets in, but I mean, if you, if you really wanted to take it to the comprehensive degree, you would get a, uh, coverage attorney in to, to give you his or her take on that. Good. Yeah. So there's some answers for the folks that are low and medium and are worried about it. There's multiple options. Uh, so, so appreciate that, Chris.

Yeah, It's late in the afternoon and you need to pay the ransom, but the attackers are saying you just want to pay the tax, the, the ransom for you. Uh, but the attackers are saying it's all or nothing. Your insurance says they will reimburse you for the payment. That means they're not gonna pay for you. You tell the IR firm, you don't have that money. The IR firm comes back and tells you the attackers have shown proof that you do have that money.

The IR firm has negotiated it down so far to two and a half million bucks. The law firm and the carrier bad grammar, there are also waiting on an answer. Um, so the key point here is, is the insurance company's not gonna step in and pay for this. So what are you gonna do? Do you know, uh, what time it is and if you can even wire that much money from your account and time, uh, have there been any changes in how you feel you should face your customers? And what are you gonna tell your staff?

They all know it's bad. And let me give a little bit of, little bit of secondary commentary on this. We, the, the, the MSPs, the ransom amounts are gonna be big, okay? Depending upon your size. But we rarely see an MSP ransomware demand less than six figures. Okay? Um, some insurance policies are very clear in that they are purely reimbursement only and they will not budge whatsoever. So you kind of need to know that ahead of time.

Um, you also need to know is if you're not wiring, if you're not typical and wiring money all the time, uh, you need to be aware of even if you can wire money electronically, meaning through a website or if you can do it in person and how much you're allowed to do the problem with work from home, and COVID-19 is, is a lot of the banks are closed as far as full service goes, and you gotta make an appointment. And chasing people down is not as easy as it used to be.

So with COVID-19, this is, you know, even more of a sensitive situation than than normal. Um, but we have gotten into like a day delay with people that had to work things out with their bank to increase their limits and that type of thing. Um, so just kind of think about that, uh, give you a little bit of extra info there to this slide as you kind of think about what you would do in this situation. Yeah, I mean, and this is scary, right?

Like, uh, no MSP, nobody wants to, to hear, I'm gonna have to pay, you know, high six figures. Like, and I have to do that ahead of time. And Thomas, I hear you, I see that in chat. Do not pay right on the money. I mean, you should never pay. Um, that said, it, it does happen, right? And there are reasons it happens. So, Chris, maybe a couple questions I wanted to lob at you.

Um, the first one is, you know, how do you see MSPs if they decide to pay and it's six figures to, you know, uh, even into seven, do they go get loans? Do they go ask their family? I mean, what do they do in those kind of situations? Any, any, uh, historical, um, advice you can give? Uh, it's tough for an MSP to get a loan in this situation.

Um, I think that there are some people out there that are maybe thinking about how to facilitate that, but, um, it's a high risk situation for any creditor, uh, looking at the situation and saying, heck, I don't even know if that MSP's gonna be in business next month. Why would I lend them the money? Right?

But yes, you see them tapping into either personal funds, uh, going out to family, friends, um, maybe, uh, we've seen it where they have other businesses and other business partner, business partners and other businesses that they've tapped into. So yeah, they've, they've had to get creative, the ones that have had to do it. I mean, it wasn't always this way.

Um, you know, when ransomware wasn't so prevalent, uh, insurance was, uh, more, more than willing to kind of step up and, and front that money, if you want to use that term. Uh, but now just with the level of scrutiny that the whole ransom payment scene is getting, you know, the insurance wants to do what they can to, um, you know, to, to, I guess make sure that it's, um, you know, the client's decision to pay that ransom and the client's paying that ransom.

Uh, Thomas gets the, uh, the joke of the day, I love it. He says, can you pause the video? We check our backups. Uh, that is the right response. Thomas, I hear you loud and clear. Yeah, that's true. But, but, uh, yeah, I mean, that's, that's really it. Uh, you should not pay. Uh, but here's the deal. You know, I think we can illustrate that across the board systemically. We've done a good job of not paying, right? Like, we're not perfect at it. Uh, but we're getting better.

But keep in mind, bad guys are pivoting. They change as we change, as we get better, more resilient backups, and we are able to recover. Bad guys don't wanna stop there. And because ransomware is not the actual threat. See, this is a huge misunderstanding. We have people think ransomware is the threat. It's not the threat, it's the outcome of the threat. And so if the outcome of the threat is ransomware and, uh, that you don't pay the ransom, it's possible that they can do something different.

So now that can be shifting to data exfiltration, and you see this so bad, guys are now saying, if you don't pay, I am going to release tons and tons of data on you and all your organizations, which then gets into regulatory issues, right? I mean, if you're hipaa, for example, you had healthcare clients that must be reported to HHS. And so we are seeing bad guys shift their tactics. If you don't pay, they're going to xFi.

And this is all because ransomware's not the threat, it's the outcome of a threat, and that outcome can change and does change. So just some things we've gotta really, um, keep in mind as we pay attention to this entire kind of attack service and why MSPs face such a growing systemic threat. Yeah, I mean, for MSPs, if your customer backups are, are gone, or, you know, we've got a number of situations where the customers have de have decided to opt out of BDR or whatever the case may be.

And the, you know, the MSP goes, well, they opted out of it. They're on their own, you know, they're on their own in that case. Um, it's not as easy as a decision like that. And so, you know, there's, there's decisions. Now what we have seen now with these prices going up and with the exfiltration, uh, the data that people are, are even less likely to pay now. They're so fed up and the cost is so extreme.

And, um, also they just kind of take their time, you know, like I talked about earlier on, where you want to, you know, kind of slow it down a little bit. Uh, they're figuring out ways to recover their data. I had a call this weekend come in from a gentleman, uh, small organization. He got hit. Um, the amount was pretty substantial. I can't remember what it was off the top of my head. I wanna say it was like $300,000. That's what it was, really, really high amount.

Um, you know, he was real eager to do it, but he didn't have that kind of money, didn't have the insurance. Uh, so I said, Hey, just step back, take some time. I'm gonna tell these guys it's not gonna happen. Uh, meaning the attackers. And he was able to do some research and look around, and he actually was able to recover things that would get him back up operationally.

Now he lost a lot of historical data to him that wasn't, that there was no legal or regulatory risk to him not being able to restore that. Uh, it was more, it was gonna be an inconvenience for a little bit. But I, I gave him the analogy that says, just kinda like you have a house fire. It's a bunch of stuff that burns up in the house, but everybody gets out alive. You know, you look down the road and yeah, you lost some stuff and you wish you had it back, but you're still living fine.

And so he, he, he agreed. And so that's a perfect example where I think A, the attackers out-priced themselves, and b, for him, he just took the time to go look for other ways to possibly recover that data. And he did. So, um, Ken, you know, we just kind of talked about this, you know, the, the, the, the one thing is just keep notice about your staff and how they're feeling. Keep a good pulse on that.

Uh, you'd be amazed at how many of those employees that you think are fantastic and always in the right mindset, uh, in these particular situations can do it a, a complete 180. So, you know, just keep your pulse on that or have somebody that's pretty good at reading people and their emotions and that type of thing. Keeping pulse on that.

Um, you know, if these guys exfiltrated data from you and they had access to your financials, so it looks like they at least accessed your data, uh, what are the chances that they actually exfiltrated from your customers? And so that's one of the things that kind of changes the ball game. And, and Wes just touched upon that exfiltration, again, if you do have to pay you, you know, you're gonna have to pay. And you need to know your financial capabilities.

This is a very crucial point of communicating with your staff and your customers. This is where I see things kind of go sideways. We see where it gets to a point where, especially owners and principals feel the need because they have long lasting relationships with a lot of these clients. To pick up the phone, call them up, and just talk their ear off, and sometimes talk their ear off and tell them things that are either speculative, premature or whatever the case may make matters worse.

So we have to be very careful with that. If you're an MSP and you've been hit this way, um, and, and you're, you, you know, you, you, you have quite a few customers under your belt, you want to have crisis communication right off the bat. So that's typically covered under an insurance claim. Most of the time the law firm is gonna be the one that brings in the crisis communication team. 'cause they're gonna bring in somebody that they're very familiar and work with.

And so you wanna bring that in right away. I mean, you don't wanna mess around when it comes to crisis communication. You don't wanna try to plan all that out because each situation's different. So in your IR plan, you want to have a step about bringing crisis communication. They're gonna want a lot of that same information that I mentioned earlier with the attorneys. Uh, but you definitely want them there. Now, that doesn't mean that they're gonna dictate a hundred percent.

You are going to be involved in those decisions. You're gonna be able to provide feedback, you're gonna be able to provide input. And yes, of course you have the full power to do whatever the hell you want. And we see that happen. Uh, not ever a recommendation, but, but that has happened. That's in, that's fully your power. Um, but, um, something to think about. And, and again, pay attention to your employee's, state of mind, states of mind. So it's very late.

You missed the deadline, the wire, that money. Uh, you've stood up a new instance of your RMM. The clients are checking in. So you don't have your RMM that before it's crashed. It's, it's, you're not getting it back. You've gone ahead and stood up a brand new RMM and the agents are automatically checking in because so many customers are work from home. The damage is not as bad as it could have been because many systems were not logged in.

So we have seen this with VPN where people, it's more VPN, not all the time VPN, but people VPN whenever they need to. We have actually seen where the damage to the actual endpoints is far less. Servers are still high endpoints are far less because of work from home. Some of your customers have their own backups that were left intact, and you've assigned people to assist with those. So you had your own customer's backups were in good shape.

You also had customers that had their own backups that were in shape. So you're still extending in olive branch, I guess, per se, to them to help them. Your attorneys want to be, want you to be the communication liaison with your clients and not have the IR firm interact with them. And what I mean by this is, when we get into situations, uh, let's just say me, a a a lot of the MSP cases I get involved in, it's very easy for me to jump on the call and talk to your customers for you.

The attorneys don't like that. Um, they think there's some liability there, even though, you know, I know what I'm doing. Uh, so just keep that in mind that they're really gonna want you or someone that's, uh, equivalent to you or closely equivalent to you in your organization, uh, to be the, the person communicating, sending the emails out, especially with phone calls.

A lot of times the attorneys, you don't want the attorney to jump on every phone call because that will put people in kind of defense mode and you don't want to do that. So they're gonna coach you behind the scenes. The customers are frustrated and you're starting to see that, but they're definitely handling this better than you thought. So did you get some sleep last night? Um, do you have any staff? Uh, oh God, do you have staff that stay, stay at the night?

Do you have everything lined up with the bank? Does your IR plan have a plan of action for day two? Or do you need to produce that now? So you need to get some sleep. Can you get some sleep knowing who you are? Um, do you, do you assign some staff to stay up and babysit through the night?

Um, at this point, do you feel with your relationship with your bank, knowing your banker, do you think that at this point in your life as an MSP, if you were in this situation, would you have got through to your bank and got things done? And, um, you know, does your IR plan today include a day two? Or does it plan or does it have some process to create that plan? Some, some questions.

While you're thinking about that, are you confident that you're the type of person that can call it a night and get some sleep? And I'm gonna tell you, there's so many people that are wired up that will go ahead and work through the night, 24 hours, 36 hours, whatever it takes. But when it gets around to day three or day four, where it's really important for you to be aware and alert, missing out on that sleep is, is detrimental to the whole process.

Um, does your staff, do you believe that your staffs can be able to come in, you know, day one, the adrenaline's pumping, they're getting things done, you know, they're, they, they, they, they're in the battle. Um, are they equipped to be, you know, a day two type team as well? Um, do you know your customers well enough to know how they're gonna handle a day? Two?

And that's a kind of a loaded question, because what we find is the customers that we think that are gonna be great on day two are the ones that are a nightmare and vice versa. Um, but if you have these credit, if we have these crucial conversations with your customers ahead of time and talk these things out and explain that, a lot of times these things are not gonna be solved in one day, two days, or three days, at least you kind of grease the skids and prepared them for that.

Do you have a day two plan? And can you handle or delegate that day two plan to someone else? Meaning you can only do so much. And so is there someone in your organization that can take that day two plan? If you already have one that's existing, it will need to be reviewed and probably tweaked. Um, and do you know how many of your customers have their own cyber insurance policies? So this is really important because the customers with the insurance policies may or may not file a claim.

And once they file a claim, you're gonna start getting called. So, um, depending upon your attorney and the IR firm you have, how those calls are handled and, and so on and so forth, uh, is not consistent. Uh, you know, for us, when we're involved and when we're engaged, uh, we, we try to take full control. We have relationships with most of the other IR companies out there.

So we can get on a phone call and kind of explain the situation and say, look, you're just gonna get in the way right now. You know, we promise you that we will share what we can when the time comes, uh, but right now, please do not try to go and talk to the attacker on your own. Don't go out there and start imaging machines and doing a bunch of crazy stuff. 'cause you're just gonna screw everything up. And in this particular case, it's an MSP attack.

So all the forensic data that's gonna be worth anything is gonna be within the, the MSP. And most of the time, uh, they'll agree and then kind of back off. Uh, if they don't, then you gotta get the attorneys involved and try to do that. Uh, but that's kind of how it handles. So knowing the potential of how many calls you may get, um, you probably won't get those on the first day, but you'll start getting more and more and more as time goes on. You need to get some sleep.

Uh, there's not much you can do at this point on this day. One, you're gonna be waiting to do the ransom. Um, you may be able to chip in and do some data restores, but you have your teams doing that. You need to have pretty much be ready to go, uh, refreshed and everything for day two, your staff needs to get rested as well. So you're gonna have some, some people that are gonna stand up and say, I'm in, I'm here for the long run.

But you need to be, uh, sensitive to that and, and, and force them to take breaks and get some sleep. Um, it's stressful. And, um, there will be customer surprises tomorrow. And that's where I was hinting at the fact that the customers that may have been in a great mood today are gonna be in a real pissy mood tomorrow and so on and so forth. And so there's no way in, in an IR plan that you can predict that.

Uh, but you can at least be prepared for the worst, uh, and, and hope for the best as they say The next day you must wrestle with the bank some more about getting your wire limits increased. You cannot avoid using the term ransomware when you're talking to your banker. It takes some discussions and this delays the wires. You, you have no idea when the wire will go out, go out. The good news is all the customer servers are either restored or are in the process of being restored.

Their desktops and laptops that got encrypted are screwed and will need to be re-image if you do not pay. And right now you're not sure you're getting calls from other IR firms that have been assigned by a few of your customers. Cyber insurance carriers, that's what I mentioned previously. Your IR company confirms. The attackers have provided proof that they have ex exfiltrated some data from 20 of your customers. And that proof contains non-public information.

So employee information, healthcare information, I mean, it's gonna run the gamut depending on what industries or industries that you sell to. And the ransom amount has been negotiated down to 900,000. And the reason for that is, is time has gone by. So the fact that this wire process has taken so long, the attackers have, uh, the, the IR firm has been able to drive down that price even lower.

So it's one of these things where it sucks that you're not able to get that ransom paid because you do want to pay the ransom in order to get other machines to basically reduce the amount of machines that you have to re-image. Uh, you also have some concerns about exfiltration that that $900,000 can buy you. Uh, but during that delay, it helps you drive down the price.

So when in the earlier days, which is not that long ago, when ransom amounts were 5,000, 15,000, 50,000, those amounts were pretty easily paid. You got the decrypter and you moved on quickly. Now that the amounts are getting larger, it's taken longer to get them paid. So the downside to that is it's taken longer. The upside to it is does give it the, the, it does give you more time to negotiate that down. Um, and so sometimes it balances itself out.

So right now, what is your plan for workstation re-imaging? If for some reason you don't pay the ransom, what are you doing about preserving your customer servers prior to the restoration? What are you going to communicate to the other IR firms and should you still pay the ransom? So I'll give a little bit of detail on this one. When you restore your customer servers, you need to be thinking about preservation before that.

And the reason I bring this up now in this particular slide is, is even though we've brought it up earlier, the MSP tends to forget that. And we find ourselves a lot of times where they've restored the environment on top of the encrypted environment, and now we don't have the forensics evidence that we need at the customer sites. Now, truth be told that the majority of the forensic, the valuable forensic information is gonna come from you and your network.

However, there are gonna be times where for the sake of certainty and for the sake of, uh, thoroughness, uh, attorneys and others are gonna wanna see forensics done on the client's customer systems as well. And so if those systems aren't preserved, that could become a big problem. So how do you solve that problem if your customers have limited resources?

So you have to do it somewhat creatively, and you have to have these discussions with your IR from your attorneys to make a judgment call on what's the best action to take. Sometimes just bringing in USB drives to copy stuff over, uh, taking snapshots, backups, whatever the case may be, is better than nothing. And doing that, you know, we'll get the ball across the finish line. So, uh, we just gotta be thinking about that.

And even though our customers are breathing down our necks, uh, to get them back up and running, we wanna get them back up and running so their calls start, stop happening. Uh, we need to be thinking about that preservation. I wouldn't be harping on this so much if it was not for exfiltration. The exfiltration is the biggest issue.

If you don't have the forensics that exfiltration is going, they're, they're gonna assume it's the worst case scenario and the cost of reporting and notify is gonna go out the roof. And so that's what's going on. So that's why you gotta have it. You can't, even if the attackers verify that they only, I say verify, if they state they only took 20 customers worth of data, that's great. That's gonna help your forensics team directionally.

But it's not going to be anything that you're going to be able to use legally to prove that your customer's data was not exfiltrated. You gotta be thinking about what if you gotta, you know, go another day. Um, what does that mean? Does that mean you're gonna have to start imaging PCs right away? Can you kind of buy yourself a day? It's probably going to vary between customer to customer. Um, how do you engage with the other IR firms?

Um, you know, you're gonna need to talk to them or have someone talk to them that knows what they're saying. Uh, again, um, as your IR firm, I can engage with the IR firms. Typically, the attorneys don't have an issue with that. Like they do me engaging directly with your customers. Are there any other tools you can leverage at this time? So this is where maybe you have other remote control tools, uh, or you have other ways of accessing your customer's environment, uh, to get the job done.

If your RMM is not up and running or it's bogged down or whatever the case may be. And the only reason I put that in there is sometimes the power of creativity is helpful at this part. And sometimes you will have customers that will say, I don't care that your RMM is up. I don't want that thing connected to my network anymore. And so you're gonna have to agree to that and then, and then hand and go forward.

So, um, and lastly, are there any other resources that you can call upon, uh, to help you out in this situation? Do not restore without preservation. Especially servers that contain so domain controllers, servers that may contain, uh, non-public information, uh, systems that could be considered a, uh, pivot point within the network or so on and so forth. You need to do something.

The IR company will help you, uh, collect that data as quickly as possible and, and, and make it to where you can get back up and running. The ransom payment may still be a good or good route because of the workstation situation. So you kind of have to do the math and we're not gonna do it in here, but you gotta gotta figure out, it's gonna take me, you know, a thousand hours to re-image me seat PCs and it's gonna take me a hundred hours to decrypt them. That's a no brainer.

Um, sometimes it's a little bit closer and just that may make the decision for you to say, Nope, I don't want to pay those a holes any money. So I'd rather take a little bit of time and, and, and go ahead and re-image machines. And a lot of times your customers are gonna say, Hey, great decrypt them, but you're still gonna re-image 'em anyway because we don't trust them. So that may come into play. Um, it all comes down to, to downtime and business interruption loss. And this is the real key.

If your customers are gonna make claims or they're gonna sue you, most of the time they're gonna sue you on loss productivity or business interruption loss, loss of revenue, whatever the case may be. And that really comes down. So you say, well, well, I'm never gonna pay a ransom. Well, let tell you, paying that ransom and getting it decrypted and getting your client up in a day versus a month, you're gonna pay that ransom. And so, I mean, that's, that's a big difference.

Um, if, if being down a day is be, is going to, you know, your customer can handle that. But if your customer's down longer than a week, they're out of business, you're probably gonna pay the ransom. Again, it kind of depends on the, the amount and the financial math that you do there, but you gotta figure it out. Also, paying the ransom can do a couple better other things.

Number one is by paying the ransom, the attackers, if you have somebody that knows what they're doing on the negotiating front, they can, um, help with the, um, they can help with the, uh, finding out what was exfiltrated. The attackers will sometimes give you a direct re listing. They'll give you the proof. They'll do those, all those types of things.

They'll do that for you, um, which will be a, a tremendous help also, even though you're able to restore, if you get a decryptor, you'll be able to decrypt those servers. And that, that by decrypting those servers, even though you've already restored those servers, if you can go and decrypt the backups of the encrypted servers, there's a ton of forensic information in there that's valuable that we would not have from a forensics perspective. And then tap into your peer work wisely.

I know we're getting short on time. Um, so the cut to the chase, the wire gets done. You get the decryptor, the bad guys say, we've deleted the data. There's an asterisk there. 'cause you just have to take their word on it. There's no proof that they can give you. Um, you're able to get your PSA back. Um, you, you're gonna figure out a way how to mass decrypt computers through your RMM.

They find out the IR firm IRR firm discovered that your RMM was the initial point of the attack and it was unpatched. And that was something that was the responsibility of your lead engineer that's off fishing. You have three banking customers who, uh, have bank regulators breathing down their necks right now. And, uh, the IR firm was able to locate some good logs from your damaged RMM, which is good for you because those logs are gonna tell the story about what happened.

Again, kind of the same questions here. I wanna get to the rest of the, this, this is the last part. Um, you need to keep tabs of your insurance coverage at this point, and your adjuster and your lawyer should be helping you with that. But you wanna make sure that if you're getting close to your max limits, uh, you know what that means for you. Uh, some policies have a, have a little bit of a kind of a, a, a way of increasing your coverage in certain situations.

Uh, it just depends on your coverage. Um, hey, legal is always gonna take the conservative approach here. They are always gonna want to be very careful in what you say and what you do. But I have seen success where the MSP has stepped up and say with, well, these four or five clients I have here, I want to take a more personal approach with the way I communicate with them. And that is your your right to do that. You just need to be, be, do so very carefully.

Um, your customers, especially those regulator ones are gonna be most concerned if there's personal information, personal healthcare information, personally identifiable information, that's gonna be their big concern. So when you're talking, uh, to them or if their regulators are hounding them, those are the questions that they're gonna be asking about as soon as possible. You're not gonna be able to give them answers right away. The forensics has to take place.

You can't tell 'em much of anything at that point other than forensics is going, it's going as quickly as possible and that you'll get 'em the information they need. Um, don't, don't just agree to IR for other IR firms demands. Um, you don't have to. You may feel pressured by your customer to do so, but you do not need to do that. And you need to have other vendors who know your RMM and PSA other than the, the primary software vendor.

So for example, if you use automate, you really should know more than just ConnectWise to help you out. Uh, sometimes you'll find people out there that are, are better equipped to handle these situations, uh, and and are more available and can work around the clock, uh, different than you can with, with the vendor. You spent the last two weeks getting machines decrypted and re-image. It has been no easy task because of the work from home environment.

But your team has press persevered unscathed. Your carrier did reimburse your ransom payment minus a deductible of $10,000. The majority of your customers are operational and continue to be your customers. Your banks, however, have chosen to move away from you as a result of regulator and board pressure. This is a significant revenue hit, but at this point it does not come as a surprise to you. So this is, this right here is fairly typical of what we see.

We do not see a bunch of customers bolting. They're kind of upset. There's a trust factor issue there. Uh, but we do see the ones that do bolt do play a part from a revenue perspective and a significant hit. So you just need to be prepared for that. Your lead has returned and knows about the situation. He came back a week ago, and you have yet to decide how you will handle the lack of patching that led to this, as well as the oversight of your own backups.

You continue to have daily calls with legal to prepare the continued discussions with your insurance and the other client's, insurance carriers and legal has prepped, prepped you for this and that this will go on for months. So, you know, with an MSP and with an incident, the legal stuff's gonna go on for a very, very long time. So, uh, probably in this particular CRA case longer than a year and maybe even two, two plus years. So just prepared for that.

Even though your insurance is stepping in, you're still gonna be the person making the decisions. You're still gonna be, you know, giving information, helping the attorneys figure out how much amount that they're going to try to settle on, uh, with whomever is trying to sue you or trying to go after your insurance. So that's just gonna go on. So just be prepared. It's gonna be an ongoing nightmare.

You know, many of your customers are truly disappointed because they trust, they come to trust you as being their, it your policy will not be renewed. As you were served notice this week, forensics has found that the attackers did take information containing sensitive employee information from your servers. So now you're gonna have a duty to notify your current employees and former employees of that.

And they've also discovered that there were only 10 customers who had, who also had data stolen, uh, versus the 20 that was first to believe. But the, in the end, the attackers did say that there were only 10. And forensics has verified that that is the case. And those 10 customers just happened to be the bigger fish like the banks and the healthcare ones is who those guys picked on. What are you gonna do? What are you gonna do about your lead?

What course of action are you gonna take with your customers who remain with you? How can you handle the legal burden and still continue on with your day job? What postmortem steps are you gonna take with your staff? And do you know what you're going to do to remediate the exfiltration risks for you and your customers? Um, is your lead completely at fault? Is this more of a cultural issue? Do you have some kind of responsibility in this particular situation?

Was your lead just overburdened with work and that's why you or she did not get to it? What's your thought on the MSP business as a whole? Do you wanna stay in it? Um, you know, the fact that you fared well through this and most of your customers have remained, does that give you, you know, better faith in what's going on? And what can you do to ensure that this does not reoccur? Um, you need to think about, I think the most important step here is the apology tour perspective.

I think you, uh, if you just think that your, your customers are gonna forget about this, that's not gonna happen. You need to have that frank conversation with them. Apologize, do whatever you can. Obviously you wanna do that with the, the customers that are not in some type of legal dispute with. Um, and then, um, you know, as you go on in life as an MSP, uh, how you handle that and wanna remain is up to you. Uh, that cannot be articulated in an incident response plan.

So I know we're, um, an hour and a half on time, how are we, uh, how are we looking Andrew, at this point? Yeah, so we're an hour and a half in Chris. We've got, you know, most everybody stayed on. We've got a bunch of questions. Um, do you feel we could wrap up here, bring Gary back and we could talk about some of these questions that are, that are, yeah, Let's do that. Um, okay. And, uh, so let me stop sharing. Gary asked me to put up a poll question. Let me grab him.

Hey, and while you're doing that, why don't we jump into a couple questions. So, um, one at the very bottom, and Andrew, if you could help close some of these out just so we can see through would be good. Um, but, uh, one question at the very bottom. Is it typical to outsource the IR work? This typically wouldn't be done by an in-house team. Exactly. Right. Uh, that is a poor choice in almost every case to do IR work yourself. Um, that's Fox guarding the in-house also.

This is complex stuff, right? If someone was able to breach your organization, they've got full access, um, chances are you're gonna be blind to some of the things they might have done. So we see this often at perch when, uh, we have customers come in. Um, we're working with an organization right now that got hit by a rather new ransomware variant called, uh, wasted Locker.

And we're still working with them in IR teams, uh, in all of that and what we're seeing and how they're moving laterally, um, where their persistence is at. Chris, you can add to that too, but in almost no case would you as an MSP wanna do the forensics work yourself. It also looks like you're hiding something. It can have the perception that you're trying to hide your own tracks. Yeah. Or PYA and it just looks real bad overall.

Wes, uh, tell people wasted locker, who the company is behind that. Uh, like the, uh, the company, it's some guy named Chris LA No, no, the organiza the organization. Right? Who produces that? Didn't you say they're like evil incorporated? Oh yeah, it's the evil core guys. Um, who some of them got arrested. Yeah. And they they've started over again. Yeah. Yeah. It could be Chris Lair. I don't know That that, that's a big deal.

Real quick on this one is because, um, because they have these guys are kind of reborn and they have this prior criminal, past, past, they may, that's one of those situations where you may not be able to pay them the ransomware because pay for the ransom because they are on a do not pay list with the government. Yeah. And that's, that's part Of it. My car. Yeah. Yeah. It's funny that there's bad guys that are on the pay list. Like it's okay to pay those bad guys. Yeah. It's funny.

Financial regulations on all this are interesting, right? And we don't need to dive into all that today. But, uh, once the federal government has identified somebody as a, as a bad guy, or if it's identified known as a nation state, for example, North Korea or Iran or someone on the sanction list, uh, you absolutely cannot pay that. Uh, you'll go to prison. It's a no-no. Uh, so yeah, that, those are big deals. Um, let's answer another question. There's some really good ones that came in.

Uh, let me scroll back over here. Um, let's see. How many MSPs are realistically able to survive a full customer base crypto incident where the Arem tools used to compromise tins or hundreds of customer networks? Obviously it would be very difficult for customers to trust you after a major incident like this. Chris, I truly know of no one in the world better and more qualified to answer that question than you.

I think you've answered and handled more MSP, what we call it perch, buffalo jump attacks, where the MSP has been, you know, let off cliff with their entire herd. You've handled more of these than anybody. Can you kind of talk about the aftermath? Yeah, I mean it's, it, they, they all the ones we've dealt with come out fine in the end. Are they the way that they, where they, the way they were pre attack? No. Do they end up having to downsize A lot of times, yes.

Uh, but they, they, they stick, they stick it out. And over time they're able to get net new business in the door and that type of thing. Uh, you know, I don't know what kind of effect it has. Maybe like, maybe the customer's not as willing to spend as much money with them going forward or willing to take them at their full trust value. But it's, it is weird. We, I mean, there's an MSP, I've seen an MSP out there not pay any ransoms and just say, Hey, customers, you're on your own.

Go talk to Chris and, and if you wanna get your stuff back, pay, pay, let Chris handle that and get your stuff back. And, and they stick, they stuck with that MSP. So it's, uh, I'm, I'm, I'm not telling you, hey, go out and get attacked and you're gonna be fine. Uh, but I'm not telling you it's not gonna kill you either. Chris, I I, I I deal with him after you leave, right? Like I'm, I'm, I'm left behind, right? I, I get to clean up. We'll help 'em clean up after.

And what's funny, and I, and I've seen some not make it or just it was too much, but like you said, many of 'em not only survive that are on top of it, um, but they actually get their customers. That's when their customers are willing to spend a lot more with them. So they wouldn't pay 'em. They're, you know, $180 a seat before or whatever.

Now they're willing to, and basically the theory is, listen, are you better off with me who just went through this or with someone else who maybe has not dealt with this left? Like, do you think it's gonna happen a second time to me? Like, look at me. Right? You know, I had hair, you know, three days ago, you know, on this thing. But the point of it is, man, that's a really risky way to get where you can get to now and have those same conversations with those customers.

So you think about, we always talk about Wes, what's involved with prevention and all the discipline today. We just talked about, forget about prevention right after the fact. How much discipline and how much has to be put into place on the response side. And again, there's risk, there's expense. And are we getting paid enough as MSPs not just to do a good job, but are we getting paid and make enough money to live with this risk that we didn't have five or six years ago?

Or a very, very small percentage. Like you should get a return on risk, man. Yeah. Gary, you bring up a good question and I want to return this to a question directly at you 'cause I don't know anyone better to answer this one than you in the chat, Steve said, how can a small MSP afford to get set up before to get new contracts, cyber insurance, and setting up all of these things in advance, right?

How is there now a barrier to entry for MSPs to get in the door because security has just gone into this stratosphere? Uh, or is this an opportunity? What wisdom would you give us, Gary? Well, first off, I'll give you the reality. The reality is, um, the MSPs don't do it all and just get set up and take clients. They just don't do it. I mean, a large percentage of MSPs not just on incident response, but on, you know, prevention. They're not set up to do it. Okay? And I'll give you a hint.

If you're out there at 130, 140 whatever dollars on average cost per seat, I know that math, um, you, you don't have an IRR plan that's well run and up to date. You're not doing regular restores of your customers, you just the, just the labor alone that it takes to do that. So the bad news is here that, um, it's neither answer. It's not that they see it as a barrier, and it's not that they're doing it.

They're just going ahead and living with all these risks and their customers are living with these risks right now, Wes, that's, that's the, that's what's happening where the rubber meets the road. Yeah, you're right. And Jason sees the same thing in Chad. I see him saying, you know, Hey, look, we're still at a race to the bottom, right? And a lot of clients are still wanting the cheapest MSP that's out there, right? And there's a difference between what you're doing, what they're doing.

I remember being on the phone with someone recently, and Gary, I think you and I talked about this one time too in a webinar, um, where, uh, you know, someone was telling their per seat margins. And the answer is what? Like, you can just say right outta the gate, if you're paying this much per seat, there's no way that they're securing you. That's not even in the picture here. And I think that shocks clients, right?

Because they have this expectation that you're providing their security forum, right? Yeah. I'm, I'm helping a friend right now who's looking to change. You know, providers, first off, trying to find a proactive provider is not easy to do. So they're sending me and I'm telling 'em what questions to ask, and I'm just, you know, and at this point, you know, they're a small firm, but they gotta be secure. They maybe have 20 people. They don't care whether it's gonna cost them 3000 or even 4,000.

They're not that concerned about the price as to they can't live with those risks in their line of business, and they're struggling to find the local provider that, that can do it for 'em. So, uh, you can weaponize your competitor's low price right now, this is an opportunity, uh, Wes, because customers, you can lead them to understand that risk so much easier today. Um, and again, being involved with things like today and what we went through, this changes your perspective.

So when you're in front of people, you have that confidence to have a conversation about what their risk really is. First, we have to know what our risks are. Hopefully today that helped. Then we need to translate that to business risk for every one of our customers. And it's real Gary. Hey, Gary. Yeah. So I, I put up a poll and, um, I think, you know, this is something that is really just critically important.

I don't sell a so SIM solution, but I do know enough about this business and working in IT for the past 20 years, um, even before we called ourselves MSPs and just working in the business trying to survive, right? And your m your, your customers run good MSPs, but it, it, they're busy. They're trying to gain new customers. They're trying to, you know, put in, uh, new tools, new processes, you know, take care of tickets.

I, I just, you know, if you take a look at the results, do you have a third party SOX sim? Like you're, I, I can't be emphatic and, or I shouldn't, let me not even say my per perspective. Chris, your perspective on the part of prevention versus protection and why, like, is it a no brainer today? For a small fraction of money. I mean, you don't have that many endpoints for, you know, for crying out loud.

Why Is there any reason an MSP today should not have somebody watching their logs and their Yeah, I mean, I mean, that visibility is key. I mean, without that visibility, you're, you're completely flying blind without, then let me get, let me give you another angle. I was talking to an MSP earlier this morning and, and I was kind of talking through some of this stuff and he was talking, well, you know, we have this prospect, they have their own it.

Uh, they want to do the, they want to be able to do some of these things on their own, and I'm concerned about that, and so shouldn't I do like a hold harmless thing? And I said, well, yeah, I mean, I hold a harmless thing, uh, whatever. I mean, what really needs to happen is you, you need to have logging enabled so you can know what the hell that client is doing. Because when it, when something breaks, when somebody opens something, they're gonna immediately point the finger at you.

And if you don't have that logging and that evidence to support that, that client did that, you're, it's, it is pretty much gonna be your fault. And you're gonna have a, it's gonna be very hard to defend it. So if you want to talk about having that logging as there as a security feature for your customer, yeah, fantastic.

But that logging and that event handling and all that is, is, is will reduce your risk if something happens in that environment, that's not your fault because, hey, we're in the us people can sue you. You're gonna have to pay the price for your defense. And so that's just reality. And then when these things happen, the, the first and biggest finger is gonna point to you every single time. So there is more than just having that kind of, you know, cliche stuff for the security stuff.

There's that evidence that you need to, to, to, to defend yourself when that happens. Yeah, and I'll just say this, and Wes maybe you can close the, out this particular poll with, but you know, a very good friend of mine is the global SOC director for the largest engineering firm, literally on the planet and 90,000 employees. They have an i a cyber budget and resources that are beyond imaginable. And they're constantly chasing out the bad guys. They're constantly in their network.

And again, for us to think that they're not doing that to us. Um, so Wes, you look at a lot of data in, in, in, you know, in, in a massive elastic platform. What, what would you say to that? And, and, you know, detection versus prevention? Yeah. Y you know, um, a few things about that. One, so everyone knows what we're talking about, right? Identified, detect, protect, respond, recover. When you look at the cybersecurity framework that was developed by NIST kind of splits into those domains.

And I think most MSPs, we've done a great job with prevention, right? We have tons of prevention eggs in the basket, and there's a reason why, I mean, if we can stop an attack philosophically from ever happening, we're better off, right? But the fact of the matter is, and one of my favorite security officers says this all the time, he is like, prevention eventually fails, and when it fails, what are we going to do about it? I mean, heck, how are we even gonna know that it failed?

This is so important for us. And that's where detection comes into play. And then if we have good detection, then we can take action before we have to do something that, you know, like handle a ransomware event. So like I say this to people all the time, don't think about these kinds of attacks like push button, get ransomware, push button, get, you know, significant breach. It doesn't happen that way. The way it happens is something happens through some kind of initial access.

Bad guys then escalate those privileges. They establish persistence, they start looking around. It takes sometimes days, maybe a week or more. That's all time that you have to detect even just one little thing that something has happened, it's not right. You know, one thing thing I love to say all the time is, you know, bad guys, yes, they, they only have to be right once, we have to be right every single time.

But the second they have access to us and our network and some kind of breach begins, that entire paradigm shifts. And now we only have to be right one time to say, Hey, something is not right here. We should take action on this now. And that is so valuable. But that goes into that detection piece, right? And then you go into the response and recover, which is what we're talking about today in this tabletop test. Those are the things that were, are difficult.

So Kyle, uh, SLO and myself from Huntress did a talk, um, just a few weeks ago on all of this. And when you look at prevention, it's typically very automated and you kind of fire and forget, but you start getting into detection, response and recover. Those get more and more people intensive. And those are the reasons MSPs don't typically want to get involved in those things is because they're time intensive and they're a resource drain.

And so we've gotta look at other alternative ways of bringing those costs down. So in the world of detection, SOC and SIM is helpful in the way of response and recovery. Tabletops like we're doing, are helpful. So, uh, Andrew, just some thoughts from my perspective. Yeah, and, and again, we're not talking, you know, for, I know for you guys, you're talking like 15 bucks all in, I think even with Office 365 log monitoring.

So, you know, it's, we're not talking in the, in the grand scheme of costs. We're not talking about something ridiculous. Gary, any thoughts, uh, for you and some more questions here? Yeah, well, I look at me. Listen, here's where we are, why this is so important. There's no scenario where many MSPs you, you can't have zero risk. Um, and depending upon the kind of customers you deal with, um, you're gonna have some risk. We're trying to close it reasonably.

And so to me what that means is, you know, as you're telling me what you think that per seat cost will be, I'm calculating what my per seat price needs to be at 70%, you know, gross margin. And so what we, what we're saying is you have to find that sweet spot, right? You have to, you know, use the micro omics math, get the idea of where your labor costs will be to do the right things. And then, you know, that'll give you an ex a budget in terms of, of where you should be with tools and vendors.

And you can prioritize those and you can find that sweet spot that, you know, you can sleep at night, your customers can sleep at night, but you're still gonna have to be prepared. We're just not, we're so far from getting our wrist down that what it really should be that we have to find that middle ground right now. Yeah. Um, but it's gonna mean to me in most cases, what I've seen with my customers that, that I think do a pretty good job with this.

They all have increased their average seat price by 30 to $50. That just is what it took to, in, to be able to invest another 10, 15, $18 per seat and the kind of things that really make an impact. Cool. So there's a question here from, uh, Cody about has there been, you know, and Cody, we, we cover, we cover this, uh, a lot on the cyber call, but has there been an increase with insider threats since covid and work from home environments?

And, uh, Wes, you wanna see you shaking your head, you wanna take that one? Well, you know, insider threats maybe not as much in, in our purview. And that's not to say that, um, uh, that's not occurring, right? I think maybe unless you want to classify, like shadow it, right? And you guys know what we, we mean when we say shadow it. If, if, you know, we're not, if we are a robust remote work from home solution, then people are gonna figure out how to do it all on their own.

And that I would put into that category of insider threat is a big deal. Uh, how do you identify those things? Well, you need, you definitely need to, to know what your users are doing, what's happening in those environments, have good awareness over asset monitoring and, um, you know, what people are doing and have a good feedback loop to know what's working and not working. So people don't want to use shadow it. 'cause here's the thing about shadow it.

I've never met anybody that's like, oh, I love solving my own IT problems, right? They're always like, Hey IT person, go fix this for me. Right? If they don't get a good answer, they're gonna go circumvent you. I think that's only expounded more in this pandemic era. So that's, that's what I would say, Andrew. Okay. I would just like to add on here is that we just see more insider mistakes.

You know, we talked about that kind of early on with somebody going and enabling RDP just to get their job done or to help somebody get connected or just, you know, work from home. They're just not, you know, updating the firewalls and doing that type of thing. So probably more insider errors than insider threats. Yeah. So, um, uh, Wes, sunny, I saw you, uh, answered sunny here about, uh, that you, uh, answered this live.

Um, but about, do you, uh, how about when you respond to state and Yeah, I wasn't sure exactly Sonny, what you meant by that. Um, I'm, I'm anticipating that you're meeting breach notification is what you're getting at. And so, uh, now let me just pop quiz. Does anybody in chat know what the breach notification window is for hipaa? Let's see. And you mean regulated, right? So some type of regulated client, I'm assuming we lost Wesley, he froze On us. The winner is, is that requirement, right?

Is what they require. Now, what you'll see oftentimes, What was that? We missed The answer. Oh, did my internet cut out? It's not so great here. 60 days is the requirement. Yeah. 60 days. And so what you will see Sunny is that oftentimes you'll see if you're serving as like a, BA, A for your vendors, what you will see is they'll say, Hey, look, we've gotta know within 30 days, or we have some partners that perch that require much shorter windows than that like a week.

And the reason for that is because they need time on their end to assess, understand what happened, to be able to report that to HHS and get on that wall of shame that's there, right? And so breach notifications, they are regulation dependent. Different regulations and different territories have different requirements, but those are things that you want to be aware of and be watching for, uh, when it comes to those things.

And Sunny, can you just confirm yes, that's what you were asking in terms of the, the response time to the state? I'm assuming, or I shouldn't say I'm assuming, could you just give us a yes or no? Yeah. And, and, and states vary. Um, it's all over the place and it's almost impossible to keep up with yourself. Yeah, that's fair. So more of the what's the regulated requirement is your point, Chris, versus what's the state?

And, and if, if Sunny that isn't a good enough answer for you, we'll take care of it after the call. Well, you're close enough by where you can we, I mean, uh, Greg, um, okay, uh, what others here, Hey, there's a fantastic one for Gary down below, and I'm gonna let him answer this because that first a little bit. Uh, but he's saying, Hey, how do you price something like perch? And even if it's not perch, Gary, it's somebody else, that's okay.

But how do you price that and keep the 70% margin, you know, do you, how do you handle that you offer outside of the package? Yeah. You know, what's the Gary PIKA wisdom here? Yeah. So the short answer is look, um, you gotta kind of go backwards and forward. You gotta really know what an accurate seat cost is for labor and tools. Then at a 70% gross margin, uh, you'll see what that price would be. So you figure for every dollar of seat cost, that's about $3 and 33 cents of seat price.

Um, then using the logic I was talking about before, you kind of draw that line where you want that core offering to be. So maybe your core offering ends up at a, you know, $185, let's see, let's say, but you have certain clients that have, you know, are regulated. Um, and so for those you are gonna have an enhanced package, right?

And so you would sell that separately, but you have to come up with, really the first step is what is that core offering that we think has everything in it labor process tools where we think we reasonably in today's environment are doing everything how that every SMB should be protected. Does that make sense, Wes? It does, yes. Yeah, and that's good. Yeah, that's good. Hey, we're still here for you guys. So more questions, uh, fire in.

Yeah, Chris, how, uh, how do you determine data exfiltration occurred? I'm wondering is it outside of them sharing proof to you? Do you, do you have any, uh, mechanisms? Is is, is that typically when you get queued in from, uh, the bad guys? So the forensics is the way that you have to do it. So the, the, so the worst case situation for us is when a client has good backups and they don't pay the ransom. Because the unfortunate side of things is, is that the forensic data is all encrypted.

And so it's very tough for us to determine exactly what exfiltration happened or what all we can do at that point is try to argue that the attacker wasn't in there long enough to exfiltrate data, or there's no, there's no, uh, evidence of any tools used or whatever the case. But I will tell you with today's technology, with what the hackers are using and the tools that they're using, it's much harder to detect that exfiltration occurred.

It used to be you would see signs of them packaging data, putting it in a zip file or tar ball or whatever. And a lot of that stuff, well now the tools that they're using to get in, grab credentials, do things. They basically have a feature that just says, Hey, you know, upload data, download data, whatever the case may be. So we, so really the way it is, is you have to do it at a forensic level and it takes time.

That is not, that's probably one of the things that people have the, the most misunderstanding about is the forensic process. That is a laborious, you know, we tried to automate it as much as possible, but it's one of those things that the investigator, they have to be right and they want to be very thorough and confident and there's peer review.

So, uh, but yeah, there's no other way really other than than the forensic side to do that, Chris, are are, are people gonna have to start doing more data identification, classification and, you know, may basically prioritizing what data they protect and how they encrypt it and protect it versus, you know, it's just one size fits all. Previously Data hygiene is huge, right? So you need to know where that data is at. You need to know, you know, where it's stored at, what, where that data is.

I mean, we've gotten in a situation recently where this company just had one massive file server for 11 different entities and it was just, they had a folder in there called, uh, everything. And so they didn't know what was in there. And so it wasn't until we actually got in there and started looking at things, we're like, did you guys know that these types of documents were in? There're like, no, it's against our policy for anybody to store those documents there.

So if you don't have a good knowledge of it, whatever policies you have written are not gonna do you any good. Yes. You need to Go ahead, Wes. Yeah, I was gonna say, and Chris, you're right, and this is coming to you as an MSP whether you like it or not, and where it's gonna come in is sort of this Trojan horse of, uh, regulation.

So you look at CMMC, for example, if you have clients that you're working with, uh, that require C-M-C-C-M-M-C adherence, you are also going to have to map in at least at level three. And there are requirements around those things, as you mentioned, of like, you know, knowing your assets and identification around those things and classification, and it's a drain and nobody wants to do that stuff. Uh, but the big works are doing it, and that's getting pushed to MSPs as well.

Um, so this is something you're going to have to, to account especially related clients. It's funny And there's value in it too. Sorry, we, you went in and out there. I was gonna say, speaking of CMMC, there was, i I the most recent, um, stars, uh, contract actually had CMMC contract language in it that they have the right to, and, and it's a huge contract that goes into small, bi, medium small business.

And it was the first time that the contract put in specifically that they have the right to put in CMMC uh, requirements at any time going forward. Uh, and that just literally hit the streets. Um, Gary thoughts? Yeah. So let, let's first off up here, uh, Andrew, Wes, Chris, thank you so much. Um, this was awesome. Uh, you know, when you see that 97% of people are saying they will watch this session for two hours, again, I will watch it again. Okay.

The notes that I took, uh, you know, from this, um, really I don't know how you can go through it and not coming away saying, Hey, we need to change the way we approach it. So, but it always gets back to the business. So I, I want to end and take a look at Dave said, Hey, 1 65 is seated a hundred seats to 16,500 a month. Uh, I have a new customer in that scenario, or paying 8,100, that sounds like a big hurdle to explain the difference. Yeah, dude, it does. It seemed like a big hurdle.

Um, today's situation sounds like a really big hurdle, right? You know, as well. And so I think part of this is, and again, I'm getting people one place who can't figure out like that is so much more, and we can never charge that. But what happens is you just start doing the work, right? And you start figuring out what's involved to do that work, and you start developing your proactive roles, your service relationship, your tool set, you get better, your culture changes.

And as it does, it gets easy to explain after you've done all that, why someone can't do it for 50 bucks a seat less, it becomes easy. And it, and it's the reason why MSPs that we work with that have the highest seat price, have the highest close ratio, and they sell the most new monthly recurring revenue because they figured out this value proposition. Gary, look to Dave's point and, and I get his, you know, initial perspective, but Yeah, I'm with you and I'm with him on this.

Uh, 'cause I have to work with people every day trying to get there. It's real How correct. However, I think there's, there's a a, a literally a chasm we have to change kind of shift when we were just IT providers and you know, you're simply dealing with a c cost around it. That's one thing. Think about what it would be like to put in a soc sim and a security analyst, right? To protect that organization.

Dave, you're talking literally, you know, on average probably the soc people that we know that are decent, $150,000 a year for an analyst, decent analyst sim on top of it. Before we even get to it, we're talking several hundred thousand dollars a year just to make sure that detection, logging, all of those things are in place. So I think it's a matter of, you know, we, we have to reframe, uh, me now that, you know, it is like they're two different disciplines.

I, it's like saying, you know, legal and accounting are the same. And I think that's what the SMB has been doing for years with it secur and it are the same, and they're not, they're massively different in terms of cost. And, and, and, and Gary, to your point, the, um, the risk right? Is getting exponentially larger.

So I think it is a matter of, And by the way, you know, that company that he has with a hundred employees, depending upon what business they're in, they have between one and $3 million a month of expenses. So relatively speaking, whether you charge 'em 8,000 a month or 10 or six, it's a relatively, still a relatively small cost if you can relate that investment to not only their cost but their revenue. Okay. Um, it, it changes that conversation. So.

Well, we ran a little bit over today, Andrew, but, um, again, this was absolutely fantastic. We're gonna make sure that we get it, um, rendered Mm-Hmm. And we'll get the link out to everybody, uh, so they can go ahead and share that. Feel free to share this with whoever, uh, you want to, um, and again, use it as a resource and this is something that we'll hopefully do, we'll get an update and we have some more things coming down the line. Yeah. We do security. Yeah.

And, and Gary, I'd like to thank you a ton. As Wes said, starting this off for getting, you know, your members involved and your community involved, that was, it was, it was fantastic. Um, we're gonna have some things like for example, coming up around ethical hacking and, you know, we're gonna have a capture the flag, our second capture the flag competition coming up here. We did it with the Cyber Con where we had 1600 people. We're gonna be doing it again, uh, come in in in August.

So we'll keep everybody in the, the know there. Yeah. And the thing that Kyle mentioned, uh, the other day, which was so true, he is like, you know, we had just in that piece of the, of the, of the, um, of the, the event, we had over 200 people compete in that. And some of the top people in there are the ones that actually found the RMM most recent RMM vulnerabilities. So these are valuable skills that we're teaching and helping everybody.

So, um, why don't you take us home, uh, Gary, and round us out, and thank you everybody, and, uh, really appreciate everything, Gary. Thank you. Yeah, just great. I, I, I mean, I don't know how you can't feel differently about where you are and motivating you and what you need to do, but, but I, I, I want to close by saying this is not about, look, some of this is scary.

I mean, there was, when you brought, when Chris, when you put up how professional that notice was, I think everyone felt a little nauseous, right? When you, when you put that up and we saw like, what we're up against. But I, I wanna end on saying that this is a great opportunity. It's just gonna take change and it's gonna take work and effort.

You have to educate yourself and have skill sets and business knowledge on the command over your cost drivers and your value proposition that you didn't need to have okay before, but the opportunity to grow and have margins and build a more valuable business than you ever had, and separate from the competition is also bigger today. So I want to end focusing on an opportunity and, and hopefulness of what, how we can leverage this. And if you're here today, that's a great start, man.

Wes closing thoughts. Hey, thank you for having us. If you had just one take home, then that was a win for us and we would love to know what that is and, and certainly share this out. This is recorded, so the true methods folks will be getting this out to you, but thank you for taking some time out today, and if we can ever help you in any way, definitely reach out. But thank you guys so much. Wes, I put the breach report link in the call to action below here that you can just, Yeah, thank you.

Makes you right to it. Chris, closing comments. And again, thanks for leading that. It was a, I know it, it takes a lot of effort on your part to prepare this. Really always appreciate everything you do and, and I, I know what a resource you've been for so many MSPs over the, you know, three years I've known you. Um, again, appreciate everything you've done, uh, in the industry, but closing thoughts from you?

No, I appreciate the opportunity and there's some more content that we'll make available that was at the end of that, that I think everybody will find informative. What I would say is, yeah, I mean, just like to echo kind of what Gary said is, I mean, this is an excellent opportunity. I mean, the, the SMBs especially are not gonna solve this problem on their own. The MSPs are gonna be the ones that solve this problem, that bring these solutions to base.

So it, you know, it's a, if you could call it a make or break moment, but the ones that make it are gonna come out awesome. They're gonna be on top of it for years to come. So, so embrace it. Awesome, Gary. Alright everybody make it a great day. Yeah, take care everybody. Bye.