Tabletop an RMM Breach

All right. Welcome back everybody to day two and live here with Gary Pika. I think Wes, Spencer, and Wes is, Wes couldn't make it, so he sent his avatar. Yeah, This is my, this is ai. Wes real, Wes is doing dictator things. Fantastic machine learning. Wes and Chris Laer, thanks for joining again. Hey, Chris. Hello. Alright. Hey, welcome all, and thanks for coming back. Thanks for a great day. Yesterday, what we wanted to do, we have two hours for this session.

We don't know if we'll use it in full, but Gary, we wanted to maybe just do a quick recap, Wes, and, and, and was gonna kind of make a few comments on yesterday. Um, should I let you kick it off? And, and anything in particular you'd like to share before we officially kick off the end of the, uh, tabletop here? Yeah, what we got good feedback. I'm glad about that.

And if you think about the three sessions yesterday, you know, in that first session, you know, talking about what cyber resilience is, and Wes like, I thought you and Ryan did a great job of taking really a lot of things we've been talking about for 38 weeks in the cyber call and in an hour put explaining them in a way that really tied everything, the different frameworks, the conversation, the way to look at things. I don't know, I thought it was really good. Now, I, I agree.

I, you know, I've, I've struggled for the longest time, Gary. It's really hard to talk with this filter on, so I'm turning it off. I've struggled for the longest time of like, how do I concisely bring all of this together? And I'll be honest with you, like, that was the first time, the week before when Ryan and I were sitting down crunching through it. Like, I finally felt like the light bulb came off. And so it wasn't like, this is like a revelation I've been sitting on for a year.

I just kind of figured it out myself. Um, but I, I, yeah, I, I agree. I thought it was really, really fun. And, and did you guys, I hope you guys that, that were on that session with me, did you guys like it? Gimme a thumbs up, uh, and chat. In fact, do this, if you guys enjoyed yesterday, just so Andrew knows, give us a raised hands or a thumbs up or something like that in chat as a thank you to, um, to Gary Pika, to Andrew Morgan for putting this on. Uh, this has been a great one day.

I can't wait for today. Awesome. Yeah, yeah, yeah. And, and then, um, uh, Chris, uh, you and Mike talking about incident response. Again, really good getting to the core, the essence and, and some of the big stumbling blocks. So kudos to you on, on that session. And then the technical session was really good. Yeah. Yeah. It was, it was.

I, you know, I've gotta tell you, Gary, you know, you've said this week after week and, and a lot of the same people that show up on the cyber call are here, but this community, I think we, you know, it's a group elevating one another. You, for years have talked about a rising tide flight floats all boats. And you know, I think now more than ever, um, this, this is, this group is helping one another do that.

And so I'm, I'm grateful for you and, and Wes and Chris to, and everybody that's put an effort into this. Yeah. We're, we're undersea man. We're not gonna go down. We're, we're, we're here. We're gonna help everybody, uh, lead the way, uh, on this thing. And the last thing I wanna say is from yesterday and today, these are things we want you to listen to a couple times because you gotta get the messaging, you gotta get it.

You gotta like, you have to be able, when you're talking with prospects, when you're talking with customers and your team, we gotta start to say things in a more succinct way, which is not easy to do. Right. Took us 38 weeks, a point where Wes, in an hour, uh, you guys could, could really kind of start to, to, to get it honed in on how to look at things and how to explain things. Yeah. Yeah. Gary, I'll wrap things up with, you know, seeing in chat, what should the all in seat price be around?

And, you know, I remember when you and I, well, it was your, it's your business. But when I first started working for you, like in 2010, and you know, you were saying things like 1 25 a seat, 1 35 a seat, and people are like, oh, there's no way. Well, now today our people are saying 2 50, 300, yeah. 300 A seat. Um, talk about inflation. But, but in all seriousness, um, we know MSPs are getting this, and it always comes back.

If you've been on the cyber call, if you've been around us, you'll be like, alright Andrew. We've heard it enough times. But I think repetition being another skill, Gary sales, I will equate sales to what we're going through right now. You've always talked about command and you know, you can't, if you don't have command of your own business and the sales process, how are you gonna go teach somebody else when you hire your next sales person? And you'd always ask people like, do you have command?

Can you do it yourself month after month at recurring revenue? The same things apply here. If you're not doing it internally, you can't communicate the same value structure and the same issues. What, what are your Thoughts on that? Yeah, listen, it's always that way, but because of security, how fast it's moving, the risks around it, having command over this specifically because it's the biggest sales wedge that we have, right?

On the new client acquisition side, but it's also, um, we have to have those conversations with customers. 'cause if they don't finance these, the roles and the process, you saw what Wes, you said, listen, when you get, when you went to the one side, it's all people, dude. Right? That's right. And, uh, you know, technology, and you hear this from a vendor, right? Technology will only get you so far and it will not get you to the finish line.

And it will not be the one thing that bails you out when the incident happens. It just won't, it's a part of it. But if that's your saving grace, you're in trouble. And that's a scary proposition for a lot of folks that are on this call in the sense of, um, you know, many of us come from small MSPs that don't have a team of, you know, 20 security analysts or even one security analysts. By the way, Craig, welcome. I see that you are a newbie here.

Just wanted to shout out and say, hey, and in a minute I'll post a link to the cyber call. Uh, so you will, um, uh, you, you'll have a link to that. Awesome. Too bad we're not in person, then we could take him out later at the night and initiate him the proper way. Oh, That sounds like, uh, Let, let's do this. Uh, everyone, let's, uh, let's get started. Yep. Um, Wes, Chris, we're gonna hand it over to you for our tabletop and, uh, we're officially underway. I appreciate that.

Yeah, so for, for those of you that, um, maybe many of you have seen these before, we've done 'em in person and, and Wes and I always say we, we prefer to do them in person, but, uh, we are where, where we are today. And so the idea behind this, when we're doing them in person, Andrew, did you have something to say? I did, just for, for the recording sake, Chris, can you just bear with me a second? Number one I'd like to, do you have an intro slide? Just ask. Yeah, I have a Slide deck. Yes.

Okay. So when you get there, just gimme a cue when you get to the intro slide, you know, I wanna queue up, let you guys introduce each other, um, and then, you know, we'll do a full kickoff. So that way when we curate this, get it into, you know, content, by the way, like a true methods portal. If you're a true methods member, this is gonna be curated for you. It'll be succinct and everything like that. Chris, sorry to interrupt you. Lemme Know. No, that's fine.

So we'll just go through this a little bit of the, in the, the, uh, informal stuff and then we'll, we'll get to that slide and you can start your stuff, Andrew. So, yeah, so, uh, typically in a, uh, in-person situation, we're in a, you know, our room started at exercise and over time it got really, really big. And so, uh, we said, Hey, look, if you're in the same company, split up and kind of group with others. So, um, you can kind of exchange ideas and that type of thing.

We can't do that here, so, so do your best here. But obviously the goal of this thing is to be interactive. And so, um, I'll have a slide just to kind of speak to that and, and a little bit, but just like anybody else that's been part of the cyber call and, and been part of these things that Wes and I have done, and we, we definitely want you to learn and in order, we know engagement is part of that.

And so, uh, the interactivity, so we'll interrupt slides, we'll do different things like that to answer questions and do those types of things. So it'll, it'll, it'll be a good time. And, um, and then we'll get started. Uh, unless Wes, you had anything you wanted, Um, a couple other things to remember, this is recorded, so we want you to go through this, learn from it. But if you feel overwhelmed, that is okay, right? Like, we're gonna get to some stages.

You're like, wait, wait, I'm not sure I'm done analyzing. Just go through it with us. No big deal. And then what you can do is use this link and do a lunch and learn, like bring your team with you next week and carve out two lunches or whatever, whether it's remote, whatever it is, and buy lunch for the team and say, we're gonna go through this thing. We're gonna devote four hours to this, and we're gonna walk through this whole thing.

We're gonna talk about what we would do, we compare to our notes, and then we will, um, you know, create some lessons learned and, and grow from it. That's why we're doing this. So just know it's recorded. And Wes just, I mean, think about what, but with what you just said, a lot of people are like, I don't have four hours. Wes, Wes, you've done a tabletop with, um, a few MSPs, a as a tool for them to get prospects and customers and things like that.

So, I mean, not to digress just real quick, Gar like, the results of that have been phenomenal, Wes, in other words, if you have a little bit of a command over what running through an incident response is, and you can walk through prospects and customers in a, you know, a a succinct manner of what, like what, what were the results of some of those tho those meetings that you guys had?

Like Yeah, you, you know, it was funny, Andrew, when we decided to do this, so while, like we, we made, Chris and I first did a tabletop at one of the IT nation events and, um, we were not sure, like I've done these professionally before. I've done them at the bank many, many times. And the first time I did it at a bank, I'm like, I'm not gonna learn anything. This is just like, you know, scenario based junk. And man, it was like the most eye-opening experience for my team.

The next time we did it, I brought my entire executive team in and we did like an executive level and like senior IT management tabletop and it was freaking awesome. And we learned so much and they learned so much. And we had, we had all kinds of notes and follow ups that we actually did follow up on and the examiners ate it up. My federal examiners, they're like, this is so great that you guys did this. Yeah.

Um, and it just, it opened my mind to the possibility and advantages of doing tabletop tests. And so we brought it into IT nation and they were like, yeah, that sounds like it'd be fun. And Chris, you remember this, we packed the house out and then we did it the next year and I think we had like 500 people in that room, so much so that we had overflow. Yeah.

Um, and so when we're, when it's hard for us to do this, like digitally like we're doing now, I mean, we do the best we can with it, but I promise you Chris and I will hit the streets again when this RONA thing is over because a in-person tabletop is invaluable, my friends. That's right.

I'd like to add on real quick to Wes saying when he was, when you're with the business side, or in your case on the MSP, when you're with your clients, they're gonna see a different side of you that they need to see, right? A lot of times they don't see you, how you react, how you behave in these types of situations.

So I know when I was at the bank, 'cause Wes and I both have banking backgrounds when I was at the bank, whether it was an incident response table test or, or when it was a disaster recovery situation, the executives are in the room with you a lot of times in those cases and they get to see a, a, a different really positive side of you. And it really kind of lends, gives more to your kind of credibility with them.

So these incident response, even though they're unnecessary for incidents, they actually help kind of strengthen that relationship. Especially with probably maybe in some of your cases where you have clients that you may not interact with people at the top, this may force those people that you don't get the chance to interact with as often as you like to interact with you and see that side of you. Yeah.

It, it's, as you jump over the slides, I'm sure Chris, I'll just say, you know, it it, Gary, it's like you always say it's about command and confidence, isn't it? And you don't develop command and confidence without practice. That's why a general is a general, they've gone through wartime for a very long time before they take over. Um, and, and we truly need it.

And senior security leaders that have command and confidence, that is so, so powerful when you walk in and you're not the chicken with your head cut off, you got a plan, you're the leader, you know what's going on, the bullets are going overhead, but you, you know how to handle this situation. It's invaluable. And tabletop tests are a big piece to, you know, getting that practice under underway. Alright. Alright. So l let's do it. Let's officially kick it off here.

And I wanna welcome everybody to the tabletop incident response exercise. Joining me, Wes Spencer CSO of Purge and Chris Lair, EVP of Solace Security. Wes, for those out there that don't know you, quick intro and then the same from Chris and we'll let you guys turn it over. Oh, well, hi there. My name is Wes. I'm a recovering bankster. Uh, I am now I was the e the CISO at, uh, ooh, my desk went crazy. The CISO at Perch, um, perch was acquired by ConnectWise.

Um, I spend my time at ConnectWise, um, uh, sewing all kinds of, uh, malcontent, um, and also doing lots of fun things like this. I think my title at ConnectWise is gonna be like VP and external ciso. So the things we're doing today is what I am all about, um, helping you guys as MSPs mature your security practices. Um, ask me lots of questions. I don't have all the answers, but just being with you and going through the trenches with you is a huge piece of what I'm gonna be doing. I think.

So pumped to be with you guys and it's your first time meeting. Um, I'm Wes, and look me up on LinkedIn. Let's connect there too. Just search Wes Spencer. Thanks Wes. And, uh, for those of you that don't know Chris la please connect with him on LinkedIn. Chris arguably has worked more of the, uh, Mr. RMM breaches throughout our country over the, since I think around 2017 when they really started kicking off Chris.

Um, Chris, we really appreciate all you give back to the community and always there on speed dial. I've gotten phone calls, I've picked up the phone at 11:00 AM at 11:00 PM at night. You're always there to help out MSPs and, and so little intro about yourself. So yeah, appreciate it, Andrew. So Chris Lair, uh, E-V-P-C-T-O of solid security. Uh, we're a cybersecurity firm that's located in Austin, Texas. Um, I'm, I'm talking to you from San Antonio.

Um, it's much warmer here than it was a week ago. So, uh, I spend a lot of my time doing incident response specifically in the ransomware side of things so that, uh, that side of, um, incidents not slowing down whatsoever. And so, yeah, so we get a lot of work that comes directly from, um, our parent company, uh, an insurance carrier as well as we get a lot of referral business.

And I appreciate those of you that call us and, and even if it's just to ask us for advice or whatever, uh, because we really feel passionately about, uh, MSPs, especially the MSPs that want to do things the right way. And, um, and a lot of what we're doing here today is to help you on that path and really to, to, to make sure that, um, you know, the Ms p industry retains its good name and its good credibility now and going forward. Fantastic. All right. The floor is yours, gentlemen.

Hey, we appreciate it. So, um, a little bit of overview of what we'll be doing today. Um, we're gonna be presenting multiple scenarios in this case, two scenarios. So sometimes when we're in person, we'll do one massive scenario, but this, I felt that, um, with the amount of time we're given and just what I see today, uh, it would be best to cover two different scenarios with you. Um, this is what we do see currently. So these aren't, um, specific cases that I'm bringing.

I'm bringing elements from different things that we see. Um, obviously the names have been changed to protect the guilty and the innocent. So that's what we're here today. Uh, we're focusing on MSPs and your clients, obviously when we're having these types of things. So, uh, we don't really have, uh, we don't come across 'cause we o operate in the SAB space.

We really don't come across many times where we're operating with a company and they're only internal it, there's typically an MSP involved at some, some level, and most of the time they are doing all the it. Uh, this will provide you direction assistance, uh, with your incident response plans. And I wanna say with that planning process, and we'll get to it towards the end, is really kind of more about communicating with your clients.

Um, it's one thing to put a plan together, put it in writing, and even do a test, but you really need to be able to communicating, communicating this, whether you're doing, doing monthly business reviews or quarterly business reviews, whatever that is, uh, you know, instant response is, is the real deal today. I mean, I think, uh, you know, you, you, you can't say we'll never be breached or we'll never be compromised, or whatever the case may be. You have to plan for those.

And the idea is the, the better, the better prepared you are, the better you're gonna be able to react and, and hopefully reduce that impact. If not, negate it completely. And, um, Wes, Wes will be, uh, interactive in this thing. He is gonna be monitoring the questions he'll cut in when necessarily he's kind of the color guy. And, um, and so we'll be going at this, so stick it on there as far as the, the questions go and, uh, we'll answer it.

Um, neither Wes nor I, um, get offended, so feel free to ask whatever you need to ask. Um, scenario one, what I'm gonna call typical ransomware. And, um, there's a reason why I'm gonna call it typical ransomware, and I'll reveal that at the end of this scenario. So it is a Monday morning. Obviously most of these things happen on Monday morning because the bad guys do their business on the weekends.

Your help desk gets the call client cannot log in, can log into the systems, but cannot access their files. You have a help desk employee named Henrietta who ask a few of the standard questions and decides to remote into the computer. It's obvious the computer is infected with some form of ransomware. It's not clear to Henrietta what it is. Fortunately, Henrietta has been paying attention to what you've been communicating to the team about dealing with these types of events.

She tells the caller she needs to immediately escalate the issue and will keep the client informed. The call gets escalated to Brad, the tier three engineer. So we can just take a minute right now and just say, Hey, here's what's going on. Uh, you get a call in, it appears in this particular case that Henrietta is doing what she believes is the, is the right thing to do.

Um, and um, and if we'll go to the next part of the scenario, Brad has been working his tail off ensuring that backups are complete. So he is been listening, he is been watching the cyber call. He knows that backups are, are ridiculously important. So he is really been busting his tail behind the scenes, making sure that those backups are rock solid. He knows he can restore the systems. So guess what Brad does? Brad's gonna restore the systems.

So the client is a backup and running that exact same day, Brad is getting virtual high fives because, you know, it's a covid times. That's the only way he can do it. He brags to his wife, you can fill into the blank of why he, uh, brags to his wife. He thinks that, you know, may get him somewhere.

And then, uh, Brad is even thinking about updating his LinkedIn profile to reflect this accomplishment of him restoring this client's network, you know, within the same day of them being able to attack the client gets a pretty outage report from the MSP detailing what happened and what, what steps the MSP took.

This is all very positive and giving, you know, given all the, the props to, to Brad and the MSP and then really showing the, you know, the client, hey, the MSP where was able to do what we said we could do and get you restored.

Now, before we advance to the next slide, uh, think about what has happened here with Brad and be, be kind of discussing if you have, if you have people in and around you or, or just in your own head with your own multiple personalities, uh, think about, uh, what Brad did here and whether, uh, Brad did the right thing. Yeah. So let's, um, give us some feedback in chat. Did he do the right thing? Did he do the wrong thing? Is it, it depends. How does it depend?

And while you're getting some, we're getting some feedback on this. Um, remember when we do a tabletop, it's not about what you should do, it's about what you would do and what you would do is what's in your procedures. And so this is not about hashtag winning, this is not about, um, doing all the right things. It's not about bragging, it's about being brutally honest with you and your team of what you would do in these situations.

And this is immeasurably valuable because this gives us the chance to go and test and correct where we may have issues. And so, for example, I see, um, Joe over here saying, wrong, lost the, the forensics. That's correct, right there. There's a, a truth to that. My big question to everyone is you on this call may know that fact, but does all of your teams do they know that fact?

This is why it's great to ask these questions to your team, maybe even to your help desk, say, Hey, if we had ransomware right now, tell me what would you do? What's in your procedures and what would be your next steps? Even questions like that are a measurably, uh, measurably valuable to, to sniff out how would they handle this and what would they do as their next steps? Because again, what you may know is not necessarily what they may know.

That's exactly, and everyone can Be honest because we confirmed that Brad is not on the call today. Yeah, Brad is not on the call. I know that for a fact. So, uh, so Brad, uh, so a lot of great responses there. Uh, you could tell people have been paying attention and know, hey, they're asking, you know, they're stating, Hey, forensics is gone.

Uh, they're stating, you know, we don't know if somebody said, Hey, we don't know what happened, what root cause, you know, somebody said, well, it is, steps may be valid. Uh, I think, uh, depending on what variant ransomware variant this is, but hey, there's no mention that Brad even looked to see what variant this was. Um, so, um, and I would kind of argue it doesn't really matter, uh, what variant it is to be honest with you, but, um, uh, going forward, let's go to next one.

So, hey, they're back up a month later, uh, has gone by and everything's good. Brad's high over his restore skills has subsided in its business as usual. But now the CEO calls, you know, her name's Cynthia. Cynthia never calls Henrietta takes a call and breaks into a meeting that you are in to say, Hey, you need to get on this call. Cynthia got a call from the FBI. Their data is on the dark web.

There's financial information, employee information and contracts that Cynthia regards as critically sensitive. How did this happen? So none of you ki a lot of you have kind of commented on this prior to the slide, but a lot of this is what you said is exactly what happened here.

We had a, we had a situation where somebody took the initiative from an IT mindset restored the client thought that was the best thing to do, got them up and running, was all happy, but did not look into seeing what the variant was and, and, and try to discover, Hey, is there, is there more to play here? Um, so I dunno if anybody else has any kind of, uh, comments on this. If not, we'll probably just go straight to the next one. So here's the aftermath.

The MSP was thinking like an IT person, like I said, you know, so this is not a slide on IT people, right? We're, we're all been it people in some shape or form. And, and, and, and the mind is programmed to, to get things to reboot, to restore, uh, to do those things, to get the client back up and running. I mean, a lot of us have been around for a long time and we were beat upside the head to do that. In today's world, you have to be very careful and true.

There may be some ransomware variants out there that are typically not known to exfiltrate data. The problem is, is we don't know if this is going to be that first case where they actually have exfiltrated data. So we have to be, be very cautious. There was no regard to this being a cyber incident whatsoever. And so that was kind of the case here.

So, um, it was basically just a total cowboy effort, even though the outcome initially sounded like the best thing, the best intent, but it was a, it was a cowboy effort.

Also, it may actually is a situation where the MSP actually looks worse because yeah, they recovered the data and everything's up, but now they have to kind of, you know, fall on the sword per se and now admit that, hey, look, you know, now we're dealing with a much worse situation because even though you were up quickly yet the proper steps were taken.

And even if they were down a day or two and maybe even three days, um, that would've been a be much better situation than now what they're gonna deal with now. And so the evidence was destroyed, no preservation was performed. Correct. A couple things that that stick out to me on this, Chris, and, and I want your feedback here. It seems like, you know, traditionally MSBs as a whole are concerned with uptime.

Like just that, that's the big concern for us is uptime and, you know, 2018 and on have caused us to realize, oh wait, security is really important too, and there's a natural conflict that comes with help desk slash uptime and security and those problems can rear their ugly head, can't they? When we have a help desk that's incentivized and judged and metriced on uptime and tickets handled, et cetera, et cetera. And yet when a security incident comes along, this causes frustration and confusion.

Can you talk just a little bit more about that, Chris, of how you've seen an incident you deal with with MSPs where that leads to some real problems? Yeah, it, it, it does, right? Because the, the around some people get caught up in SLAs and doing those types of things that they, they lose focus of, of, of the long game, which is really what's in the best interest of the client. And so we are seeing this more and more, and that's why I put this scenario in here.

Now, this, this scenario is fairly similar to, to, to a single scenario that we had, but we're getting into a lot of cases now where, um, MSPs are just a, a, they're either doing this and not taking into account any of the things from an IR perspective. They should, or b, they believe they can kind of embark upon this IR journey and they think they're doing the IAR things, right?

And they aren't, for example, we get a lot of times where they go, yeah, we preserve things, we have this, the, the servers backed up. Um, the bad servers are like, great, what about the workstations? Oh, well we wiped all the workstations because they were really important, we needed to get 'em back up. And the problem is, is one or more of those workstations could be the entry point. It could be what people term as patient zero.

And so, you know, again, there's just this, not this, this, hey, stop and pause there is jump, react, get the client back up and do those things or do IR and, and do those things. And the other thing about when you embark upon the IR is it can come across from an optics perspective is that you're trying to cover your own tail end on this, right?

When you are the MSP and something bad happens and you're kind of doing the whole work, um, who's to say that that work can be, that you can defend that work as if you were not the MSB, that there was a clear separation of what you were doing from an IR perspective versus what you were doing from a managed services perspective. So it's really complex. Go ahead. Hey Chris, You know, having, you know, run a couple MSPs and worked with a few, um, couple, a couple things come to mind.

Number one, uh, you know, this goes to the core of the culture, which is the clients down, they have to get up, there's a problem, we have to solve it. So without a tremendous amount of training starting with an IR plan, that won't happen. And also the communication needs to be different. 'cause soon as that happened, right, somebody had to be communicating with Cynthia.

Like, so, so many things are different then everything we've done our whole lives in terms of our motivations, uh, this is a big culture cultural change. Gary, can I ask you a question to your point right there, does that also invoke hey, A-V-C-I-O, which is a big piece of what you train people on that hey, now that we have an incident response plan, Mr.

Client, let's talk about if we feel there could be something going on where response, you know, there, there may be a elongated period of time and don't take that as a bad thing, setting better expectations in that VCO role. Is that something that you're Kind of Yeah, absolutely right? Setting the expectations, you know, what we'll do after they go through the whole exercise, maybe at the end we can revisit that. Look back on this and, and, and talk through that piece of it. Really good stuff.

Yeah, and the, the other thing I wanted to, to mention about, uh, and going into these, these other areas, um, you know, obviously the evidence is destroyed and I'm gonna get into a little bit more in the second, uh, in a second about that. But, um, no attorney was contacted and this attorney-client privilege is, is important.

I spoke about it yesterday on the one with Mike, but really what I wanna say on this is it used to be where, um, it was pretty standard for if you did any work that the attorney client, the attorney came in a day later, a couple days later, whatever, that all that communication was retroactive and covered under attorney-client privilege.

But there have been recent cases where that has been thrown out from a, by a judge and the all the information and communication that was done prior to the attorneys coming on was actually actually discoverable. And so this is really important.

So when you're starting to do these things and you're doing them kind of prematurely and you're not, you're, you're, you're thinking you get some steps done and you're not taking account that the attorney's not there, you're going, you're kind of not doing what's best for your client. I mean, I was on a call, um, and you know, there was a person as an example said, Hey, look, we just had this conversation over the phone.

Can you please write up everything that we said in this conversation and kind of put in the notes? And my response is no, because that stuff could be discoverable. Um, so I'm fine with verbally communicating things to you. Uh, I've been on calls where people have started a, a call like this or like a, a teams call or, or Zoom or whatever and they start to record it and I tell 'em, no, don't record it.

And so those are the things that, you know, just, if you're not doing these things every day, you're just not kind of wired and, and, and practiced and rehearse to do so. And so that's where you can get yourself in, in a bit of trouble. Now, sometimes we've seen where somebody even has maybe not a breach attorney involved, uh, but some attorney has at least engaged to at least cover that client attorney client privilege initially. That's fine.

But if you just are going at it, and the other thing I want to note on here was when I talked about how they got that outage report, well guess what? That outage report could be, um, translated as a, as a forensics report, even though it's not.

And so it now becomes kind of, hey, if you're saying, hey, it came in this way or that way and you really didn't do any forensics, uh, it's not hard for an opposing counsel that's suing your client to take that thing as and to argue that that's, that's realistic. So whatever you put on there, wrong or right could be used actually against your client in those particular situations. Andrea, did you have something to say? A que question out there, Chris?

And I don't know if it's something we wanna hold off on, but lemme pose it to you and Ian asks, how do you handle client in the case? They just desperately need to get back, you know, back up and running something we should talk about now, Chris, or Hold off. No, yeah, we can talk about it now. That's great. So, so one of the things is, is, is a lot of times we get in situations where the client's infrastructure just does not have the resources available to preserve data.

And so, you know, the the, the decision is made, Hey, we just need to get up and running. The client kind of pushes, whomever's doing that work pretty hard to get that done. And so boom, stuff gets overwritten and it's done. It's not really that hard. And if you go through, uh, explaining these types of things to the client, even if you can back up those servers and, you know, create a image of their VMs put up to a USB drive, there's a number of things that we can done.

Um, you know, from a forensics perspective, there's some tools we can tell you to run, uh, that will grab all that stuff. And if you just kind of are able to be in a position to explain to the client the importance of it, and it may cost a couple of hours, um, then you gotta do it. Now we do get in situations where it's 24 by seven and those types of things, and in those cases you gotta kind of think out of the box.

And I, I was gonna throw this term in there, uh, but I didn't, but it's almost like, um, we, we maybe should coin this term and no one better steal it, but kind of a minimum viable recovered environment. So an MVRE is maybe what we wanna call it. So, um, you know, Wes had spoke about earlier about A BIA, but really you're from a client's perspective, a lot of times there's just one or two things that they need up and running.

And so if you can figure out what those are, and sometimes it's not even everybody, it's just a few people, a lot of stuff that that we handle comes down to being able to cut payroll as an example, right? So if you can get whatever they need to do to cut payroll, you can buy yourself some time. And that means buy the efforts and time to do the forensics collection and get that done because, um, it can, it can be disastrous.

Now I can see that there's times where you just have basically the, the virtual gun to your head and you have no choice. But as long as you have an attorney there that's articulating the risk to the client that that's not a good decision to make and the client still makes it, you're fine. But I would definitely have a third party witnessing that conversation for you. But Chris, here's the key to it. This conversation shouldn't be happening at a breach.

This conversation of what would happen and how they need to be paired based on that with technology and timeframes should have already needs to happen well in advance. 'cause when you have it after the breach, it doesn't go well. No, that's exactly right. I mean, it's almost impossible to swim out of that type of discussion, right, Gary?

I mean, I mean, you can dance all you want and it, you might get a smile or a somewhat little bitty smile back from your client, but as you walk out the door, he or she's gonna be pretty p****d off and for, for the long run. Um, one thing I would add is your friendliest, nicest, kindest, longest running clients can become bitter enemies through an incident. Uh, or even just short term. I've seen it happen.

Uh, at the bank we had a very, very, very long-term customer and they had a breach on their own machine. Um, long story short, it led to, uh, account compromise and it led to some a aach H fraud is, uh, just the way I'll say it, and you better believe they came at us guns ablazing. Um, that was not a fun situation to be in. Uh, clearly it was on them and we had to have some contract language shown to their faces that they signed to prove it. But it was not a fun situation.

And my point is this, to your point, Gary and Chris, this is why we must have these conversations ahead of time and we must have our agreements and documentation in in front of us to be able to go this direction. And even just like words of advice, like, you know, if the CEO comes to you of your client says, I don't care about this incident, we're losing a hundred k, 500 KA day by us being down, blah, blah, you're gonna get it up and running.

Um, you've got to know how to stand strong on this because if you back down in a situation like that, um, all of a sudden, and Chris, I want you to add to this, insurance can go out the window in terms of what's coverable. Um, all of a sudden when you get into the legal situations, you're the one that made the changes and destroyed evidence, uh, that is going to hurt you down the road.

And so you've got to know how to stand strong and some of those things and say, look, I know this is horrible, but uh, we have to follow, um, what the law requires here and we must follow a chain of evidence and we must do this correctly. And that is why, to your point, having legal counsel on your part to guide through that process is extremely helpful. Um, Chris, what would you add to that?

Yeah, and I would say one of the things that we've seen even happen more is, uh, employee related issues, right? So something that an employee might have done or suspected of doing, um, for chain of chain of custody is so important there. Um, and it's really, you have to have, we, we always say, Hey, find a, we find a local resource to to, to do that, right?

Because, you know, mailing stuff and shipping stuff and all that kind of stuff, we don't want to give an attorney, um, you know, any possible gap to, to basically use to get, get their defendant out of a situation like that. So again, I mean, you gotta know these situations and you're right. I mean, you gotta, you you gotta play it by the book. And it doesn't hurt to ask questions. I mean, you know, people like myself and attorney breach attorneys we're, we're on all the time.

I mean, there's the, the, the law firms that we work with, they're on call. They got people on all the time, they'll jump on calls at any time to have these conversations or whatever. That's, that's what they're doing. So, um, there really, to me, there's no excuse not to at least ask those questions and be able to communicate or, you know, have somebody else communicate to your client directly. So you're not, you're not, you know, messing up the message.

There's no man in the middle type situation, no hearsay. They're hearing from a, from a, from an expert and, and why they should do these things or why they should not do these things. And, uh, we touched them on these things.

So, um, you know, I, you know, I've seen things where, um, you know, people aren't calling the insurance right away and, and, and many days have passed and, and a lot of policies have that term, that term, those terms and conditions in there where they have to be done in a set amount of time. So you don't wanna put your client at risk of, of not being able to file a claim if they can. If if after All that, Chris, the client's adamant you make 'em sign like a release and doesn't even matter.

Yeah. So that's a tough one. You know, that's, that's almost, um, a legal deal there. It's, you know, one of those things is, you know, how much if, if you're in a situation like that as an MSP, how much more do you wanna be part of that situation? And are you willing, even if you do some type of hold harmless or some type of release, you know, is that still enough? You know, they still can sue you over that, right?

And they can still claim that they were misinformed or miseducated by you or tricked in up signing that hold harmless. So there's a, you know, a bunch of stuff that could officially happened. Yeah. Andrew? Yeah. Chris, one last question then we'll hold off on questions and let you, and we continue. But I think it's a good one that Dennis brings up. He's like, Hey, what have the clients compromised through no fault of the MSP, right? Mm-Hmm. Is the SP gonna be dragged into court?

And, you know, like, is, are they part of this whole equation? And I guess it depends on does it, does it make a difference in terms of how they get involved? You just had this situation with one of my customers. Yeah, exactly. So, um, it depends. So I would tell you that, you know, when we started down this path that from, um, a carrier and legal perspective, a lot of the MSP's part was somewhat ignored, right?

Um, you know, getting into trying to sue an Ms P or do anything of that nature just wasn't that attractive of a, as a, of an option for them back then. Uh, but as the, the, the claims have grown and then as the payouts have grown, and I, I'm just speaking from this from an industry perspective, and this is, this is well known, not type of secret information, you know, money talks and as soon as these dollars get, get big enough, then the MSPs are gonna get pulled in.

So, um, you know, don't be surprised if, for example, I think somebody's mentioned that in somebody in the, in the chat, but let's say for an example in the past, uh, ransomware happened, the bad guys deleted the backups. Uh, so the ransom, the, the, the amount had to be paid and all that kind of good stuff and everything was taken care of, and people rode off into the sunset. Well, now the questions come up, well, what, to what degree was that MSP responsible for that backup?

And so those questions are starting to be asked now. It's like, was that backup being properly configured or were we getting cases where the backups are actually there but they haven't been monitored correctly or none of the, a lot of the critical data's not made it to the backups. 'cause some, somebody replaced a server and forgot to add that to the backup job or whatever. So don't be shocked that that, that a carrier may come after you.

Um, and typically what's gonna happen there is they're gonna do what's called subrogation. Uh, similar with the dudes in a, in a, in a car accident where it's gonna be insurance going after your insurance, and then obviously if there's not enough insurance there or, uh, your insurance policy doesn't cover that type of situation, which I can't really comment on 'cause there's a thousand different variables at play there.

Uh, then it could get into a legal issue, especially if the, uh, situation's big. So like, you know, Gary was referencing something, think about if you have a client that is in the five to six figures per month of MRR, and they are everything that you know, that, that, that's a great client. You don't wanna lose them. You've done everything to keep them. But then something like this happens and it's just you were doing everything right except for there's a question about the backups.

And so, and we're gonna get into that in a second, but, um, that, that loss because of how big that client is, and let's just assume with the contract that big, that that con that client is a retail operation that has to be up for the majority of the time and every second they're down is very punitive as far as sales goes. Y yeah, it's gonna be big.

So yes, you're making great money from an MRR perspective, but you really haven't weighed the risk of something bad, uh, or something going sideways and, and that being done. And you may not even have the right coverage, uh, to, to cover that steep of a loss. So let's do this, Chris. I'll hold questions. I'll let, let me let you get through a scenario here. Just be patient. If you have more questions, pop 'em in to ask a question.

Wes, you and Chris take it over and, but I think it's really important banter that we're having here. It's all very relevant, so thank you for that. Can I, can I, um, uh, we got a poll question up that I had, um, Andrew asked him to, to pop in there. It's the second one I wanted you guys to go take a look at that. Does your help desk documentation clearly delineate when a possible security incident begins?

In other words, if I am a tier one help desk guy and I just got hired two weeks in and I take a call from a client and they're saying a series of things to me, what they see, do you clearly delineate, Hey, this is actually, this goes a different route. This doesn't just go in the help desk, fix it route. This goes in a, I need to document this and trigger a series of actions that begins some kind of incident response. Does it or does it not?

If you say yes, totally cool by me, I recommend you test it. Um, there's a lot of ways you can test it. You could simply just do a role play kind of thing. You could act like a client. There's a number of things you could do.

Um, but that, this is a good question because again, we can say all the things that we just said today, but if you don't have something that clearly delineates when a security incident begins, you can't expect your downline tier one, tier two folks to properly interpret all of this. They haven't been through the experiences like you have.

Um, have them watch this tabletop, um, make sure that you have it documented of what they're supposed to do, because we can't expect junior folks to always do what we think they should do. Chris, I think you'd agree with that. You've probably worked a series of incidents where honestly, you never would've been called had they had better processes down line for the new folks and the young folks to be able to make sure they follow things correctly, right? No, that, that, that's correct.

And the one thing that, um, I forgot to mention and why I called this a traditional ransomware is because, uh, you know, the, the M-S-P-I-I was asking, I was like, well, why did you know? Why did you just go ahead and restore? And they said, well, we just thought this was just kind of a traditional ransomware case, and that's what we did. And I said, there is no traditional ransomware case, right? I mean, it's, it, it, they, they, they, they all have to be addressed the same way.

I mean, there is just no lightweight type ransomware that you can just kind of ignore or do anything about in today's world, especially from a legal side, they have ramifications. And so, you're right, Wes, uh, a lot of times if people had just had done the right things the right way, um, or at least had some chain of command in place that people strictly adhered to, to make sure that the wrong decisions weren't being made, uh, they'd be in a, in, in, in better shape.

Um, so one thing, uh, that we wanna take just a commercial break between, uh, the, the scenarios and just talk about logging and I, I figured this would be a good time for, for any of us that, that are here presenting on this to talk about, to talk about logging because it's become so much more important, uh, in today's world than anything else. And really, it kind of comes back to what I told, talked about in the scenario is about exfiltration.

And really what I want to get into this is, it used to be when we had ransomware cases, exfiltration just did not take place. But from a legal perspective, we still had to prove that out. But we weren't under the gun. Meaning like we could take care of the event, we can restore, recover, do whatever needed to do to get that environment back up.

And we collected all the evidence handed over to the forensics three to four to six weeks later, whatever the forensics report would be done, it would demonstrate that there was no indicators of exfiltration and we're done. Well, in today's world, exfiltration is the norm. And so in today's world, a lot of times we gotta get to those exfiltration answers very, very quickly because decisions have to be made.

And a lot of times decisions may have to be made on whether or not to pay an extortion amount or not. And uh, it just depends on a lot of things that, that, you know, I could go into for hours. But the problem we have is when we get forensics people involved, they can get involved very, very quickly, but they're analyst and they have to sift through a mountain of data to even get to where they need to get.

And so if you have a threat actor, uh, that said, Hey, look, you got 12 hours to make a decision, or we're going to jack up the, the price or 24 hours or whatever, or you're dealing with that, you're gonna have to, your, your client's gonna have to get money out of an account and do wire money in order to do all this kind of good stuff. And they typically don't do that. And there's all sorts of moving parts that need to be figured out.

Um, you got the volatility of Bitcoin going on right now with prices jumping or going all over the place, and for the last month or so they've been rising, rising, rising. So every minute of that clock ticks it could be 4, 5, 10, 15, $20,000 difference, uh, by the time that that original amount was stated to what it's worth now. And so it's so key.

So that's why I really want to talk a bit about logging and how, you know, we've, we, we, we've said what you gotta kind of have in your package offering today and, and why you gotta price things the way they need to be priced.

And, and a lot of times, you know, when we're talking about, you know, whether it's a solution like perch that can collect that information, um, why it all, it has to become part of a norm now because without that data, uh, the, the, the forensic guys just have to do so much more work. And, and Wes, I don't know, you know, you guys obviously built a product and the solution around this, so, you know, I would like you to talk to it as well. Yeah, I mean, it, it's a lot of work.

We end up working. We we're, we typically are not that public with, um, incidents that we're helping with, but we're, we are on them every single day. Um, and they are, uh, some have happened like three or four months ago and we're still assisting with those things. Um, they can be a drain that you had no idea. They can be challenges that you simply don't have the capabilities for.

Um, they, the amount of forensics work and research and legal process and going back and researching all over again is critically important. I will say having the data in front of you from the beginning is immensely important. It is a dream for us. So keep in mind, hear me when I say this. At no point will perch ever apologize when you get ransomware. It's not our job to stop the ransomware. It's our job to document what's happening.

And in many cases, uh, I my preference to tell you, we're seeing it while it's happening. Uh, but not even then will we always see it. That's just a fact of the matter. I will tell you, the advantage you get when you have proper logging in place, be it it per or anyone else, is the advantage you get of being able to go back relatively quickly and understand the story of what happened when it happened, what was affected, when it was affected, and how it was affected.

And the more data you have in front of you available at your fingertips to see those things, the, the easier and quicker it makes it for you to deal with some kind of incident. Um, those things are just facts and, um, you know, getting, going back and just assuming that I can get that log data later is a, um, is a, is a, uh, a trial of, of, uh, thinking that I would not want to go through. Um, bad guys know how to clear log files. That's part of their SOPs.

Um, they're gonna, they're gonna wipe that stuff right away. So, so these are things you need to think about that go into your security decision making processes. And by the way, that, that's why logging is in, uh, CI S'S implementation group one. It is the only DI don't know if you guys saw my, I didn't really dive into this very deeply yesterday, but it is the only detective control that is listed in IG one in implementation group one and getting started. Uh, think about that for a minute.

That's the ci IS folks talking about how critical log data is for us and making sure that we have it available. It is, um, you know, Wes and I are recovering bankers. We've been, do the logs and the requirement to log goes back what, only 20 years? Yeah. Um, it, you know, and yeah, we would love to, to log everything, but the stuff that really should be logged is, is kind of a no-brainer.

And, um, it's, it's, I would love to see, you know, a great robust tried, tested solution in, but at least something. And then you'd be amazed at how many things we, how many times we come across things. Well, not even the firewall is logging or not at all, or it's a short period of time, or, or they haven't checked all, you know, you go into firewalls gooey or whatever.

They have five or six check boxes, uh, to what, what needs to be logged and, and they're not checked or the one that we don't need is checked. You know, like system stuff that doesn't even matter, right? And so it's just, it just doesn't make any sense. Um, you know, antivirus is the same way.

I mean, if you're using an antivirus solution, and I'm not gonna call any of them out, but if you're gonna use an any buyer solution that doesn't have kind of some level of, of information left in it, uh, for, for somebody to go through. 'cause a lot of times our, our forensic guys will want to access that portal and find out what the, what the AV saw or didn't see, or whatever, what was logged. I mean, it just doesn't make any sense anymore.

And so I, you know, I don't wanna, I don't wanna say, you know, I don't wanna start cussing or anything of that nature, but it really is no excuse. It was like before where, um, you could say, okay, you could give some of these, uh, MSPs a little bit of a break early on when their RMS were being attacked. But everybody in their brother knows that rms are being attacked now. So there's no excuse. And from this logging perspective, there really is no excuse.

Um, you gotta figure out how to make it work. You gotta figure out a solution or solutions. You may have different options for your different types of clients that can only afford one or something, but it really is on you. Maybe not contractually in black and white, but it's on you if they're leaning on you as an IT provider to have that logging component in place and to make sure it's properly configured and working all the time.

Um, again, if, if we had some of this stuff at our fingertips, I mean, we do come across some cases where they have stuff involved and it's extremely helpful, but if we're able to access that logging, uh, it might not tell us everything we need to know about X bill, but it points the forensics person in the right direction and that saves tons of time. And you're talking about going from looking for a needle on a haystack to looking for a, for a needle on a, you know, on a kitchen table.

Um, so it's a, it's a, it's a big difference. So, um, what do you look for in logs detect X bill? I mean, how do you detect normal data extraction as part of business supposed to going out door, the door inappropriately? Well, a lot of times the log data can show us, uh, accounts that are being used in maybe accounts that are being used a lot more or differently than typically. And then we can see what those accounts and then start tracking the activities specific to those accounts.

And then we can look at, um, forensic artifacts on a server, uh, and, and focus in on what that account or those accounts we're using. So, uh, a lot of times it's not saying, Hey, there's nothing in the log string that tells us this is being exfil, but there's usually information around sessions and times and stuff like that. A lot of times it comes down to somebody says, Hey, um, um, John Doe we, we see that John doe's account was used to VPN at three o'clock in the morning on Saturday.

Oh, okay. That's, that's great. We've seen that in some kind of log or something in the VPN, then we can go look at the other logs and look for John Doe and what John Doe was doing in that particular time. Uh, the first question we asked was, John logged in at that time or working and 'cause in covid times, it's, uh, it's not as clear of an answer. People work at all sorts of weird times now. Uh, and they say, no, John wasn't working at all. Great.

Then we, we kind of, we go through and peel that stuff and then we can kind of see, we can go to that and start putting that timeline together more quickly. And, and I would tell you with, with exfiltration, and I think Exfiltration has a lot to do with it, people are way more stressed out today about these ransomware situations than they have been in the past. So they really want answers very quickly.

And it used to be they were very patient about the forensics process and they understood how long it was gonna take, uh, but now they're not so patient. And so even if the forensics person's able to go in there and start to see log data and we can actually communicate, hey, yeah, we've seen some session information, we've seen the activity in logs, we know where we're going. That can kind of, can help kind of calm down, uh, the victim in this particular case and, and, and be very meaningful.

So, um, you know, I'm not a forensics analyst. I don't pretend to be, um, those guys, uh, and girls, they are, uh, amazing individuals. They're very analytical and they just will sit there and, and carve in and, and stay on one thing for hours, days on end. And I completely respect that. But, um, um, you know, what they can tell and what they look at in logs in detail, that's about as much as I can share. Good stuff. All right, we'll jump in scenario two. Yeah, look at that. Man.

I gave you some very friendly numbers there to look at. This is a co-managed situation, right? So, um, seeing a lot of these start to become a little bit more apparent and, um, I'll talk about one, two in a second, but your help desk call this time Jeff, John takes the call. Um, the lead network engineer Vaughn and the, the lead network engineer at the client calls his name's Vaughn. He calls and states they have been hit with ransomware, been working on it for the past day.

He is pretty miffed that you had not recognized this since you monitor their systems. However, he is gonna put that aside for the minute and he just needs your help in restoring the data for backup. Uh, this is a lot for your help desk guide Jeff John to take in. He's 31 years old, he lives with his parents. He rides his long board, long board to work and he really isn't used to this much stress in his life.

Uh, he, so he reach out and contacts his service manager, Yvonne, which is kind of weird 'cause you have to mention Von and Yvonne. Now in the same sentence. So we have Yvonne vn and Jeff, John all in this particular situation with this co-managed deal. And I'm gonna stop here and just kind of talk about this because these co-managed deals, you know what I hear from a lot of MSPs from an operational side, they are a great relationship.

Meaning that, and this is really what kind of my personal opinion about things. When you have an organization that has internal IT resources, they're a little bit more, in my opinion, committed to doing things. It, they're a little bit more aggressive about doing things, which is very positive.

So you usually see when I talk to MSPs that have co-managed relationships or whatever the case may be, uh, they really seem to do more project work and, and just bigger and better things with these particular clients. 'cause they just seem to, to be a little bit more in touch with, with it. So usually the feedback I get from MSPs, again, from a revenue perspective and a relationship perspective, that these are great situations.

And the reason that I'm bringing this scenario in is they are fantastic situations, but when stuff goes sideways, they can be completely opposite of what you would, what you would want. So again, there's the trade off between risk and reward on, on these types of situations, situations. Um, so Yvonne, she's a very seasoned person. She knows how to deescalate situations well, you know, that's one main reason she's on the, she's the service service manager because she does this real well.

She starts asking Vonne some questions. He doesn't know really how it happened, but they restored their systems on their own from some online backups. And the attackers just got back in and re-encrypt. But they also found those local backups, they missed them the first time, but they found 'em the second time and destroyed them. So now they need the secondary backups.

And, um, he also tells Yvonne, Hey look, uh, we have a compliance person, Dwayne and Dwayne's gonna be contact contacting you because Dwayne is leading the charge with their cyber carrier. The reason I put this in here is, is typically when you have a co-managed situation, the client is usually bigger and more mature, and they're usually gonna have other players involved outside of it.

And there may be a compliance person that you have never dealt with before because they don't deal with IT compliance. But now that this insurance lever is pulled, now they're gonna get involved. And that's a different personality and someone that you didn't even realize that was gonna get involved. So this was back to earlier when Gary was talking about these tabletops and talking about getting those people and individuals involved.

When you go through these exercises, you can help identify these individuals that would possibly be participating in these things. So you have a little bit of warning and maybe you have the, uh, actually the opportunity to actually get to know Dwayne, uh, versus getting to know Dwayne in this type of situation.

Um, I can tell you, uh, compliance people are very nice outside the office, but compliance people are compliance people typically for a reason is they're very rigid, they're very black and white, and they like things in a particular process. And I can tell you, when you're dealing with an incident response situation and a compliance person, usually those two worlds do not kind of meet eye to eye. They're very, um, they're very operational and, and very methodic.

And when you're trying to recover systems and do forensics and do all sorts of stuff, they want you to explain everything in detail, document everything in detail, work together on a plan, and all that kind of good stuff. So if these types of individuals, um, exist in an organization, you better find out if they are, um, Chris and I would say the client, even if people did something in this case, they'll still blame you for that. That's Exactly right. What to do or not do when this happens.

That's why you're there. That's why you're there. And we're gonna show that you're gonna, it is a perfect segue. So Yvette tells Von that the backups are there and that you, that you know, you as the MSP can initiate the effort to start that process. Since these guys are co-managed, their environment is large, they have 20 servers, 40 terabytes of data, uh, their backups have been completing fine. The daily reports show that they are there. So Yvonne is confident that they're there.

So she does what she's the, the logical next step. She reaches out to the cloud backup provider and explains the situation. The provider explains that the estimated time to recover their data is 47 days. Um, Yvonne is shocked. Dwayne calls her and says, Hey Yvonne, I need you on a call in 10 minutes with a cyber carrier, the IR firm, their local attorney, a breach attorney, a PR firm, and the CIO who no one really knows too well. And, um, we find this a lot, right?

When the, um, you know, typically the MSP is dealing with a network manager or, or maybe the network people or the help desk people or whatever, the CIO kind of stays out of the mix. The CIO kind of avoids those types of conversations whatsoever. So again, kind of like Dwayne don't know Dwayne too well, now you bring in the CIO in which really don't have too much of exposure to either, and now you're getting pulled into this phone call right away.

And so, um, so what happens is these things happen really quickly and in a co-managed salute. So like if this was a co-managed situation, you would be in control a lot of this, right? But now that this co-managed and you have some different players on the inside that are making these calls, you don't necessarily have the same insight and visibility in the situation that you would have. And so a lot of these things are going on behind the scenes that you have no idea.

And so you, you've really gotta be, be prepared. And again, the reason why I'm bringing this forward is, uh, the fact that you really gotta understand the complexities and the relationships in these situations. Because again, things could be fantastic from a revenue relationship perspective, but when things go sideways, they go sideways. And I'm sure Wes, you've, you know, and all the stuff that you've seen, you've, you've seen a mix of all these types of relationships with providers and such.

Oh no, never actually, um, new to me, no, I, this is correct. Um, I've been on calls, uh, where Perch is representing, um, and you're on with legal counsel, you're on with insurance carriers of multiple sides. You're, you're, uh, with barking lawyers. Um, yeah, it's, uh, it's a harrowing experience. It's fortunate for me, um, we've not done it on the perch side, where per is the one dealing with this, uh, very fortunately not yet. Um, me just say that.

Um, uh, but this is a harrowing experience for anybody to have to go through, right? And, uh, again, Chris, as Chris has mentioned over and over and over, there's a lot of money that's on the line, and that is what makes this such a big deal, um, especially for business owners that see their life flashing before their eyes. Their retirement was their nest egg was their company, and they just, in their mind, they see it just gone in, in, you know, in a snap in the blink of an eye.

And, uh, these are things that compound the stress. Um, so Yep. Yep. Just adding a little, uh, fear to the whole thing. Yeah. And you Yeah, I'll, I'll double down on that, right?

The, the one thing that's not in here is, uh, Yvette, uh, you know, informs your, your client facing teams, what your cloud backup provider is, and once you, you tell them the time's gonna take you to recover, those end customers start calling the cloud backup provider directly to get answers, and they can't talk to them because their relationship is with Yvette, not with you. And your customers get even more answers. That's Exactly right.

And so none of that, especially when you talk about, you can call her Yvette, Yvonne, whatever the heck you wanna call her, but the, the, the point Ryan makes is a good one. And the, you know, and the fact that this wasn't really disclosed. And so from an MSP's perspective, the way that we've presented our services to customers is, Hey, we've got this stuff, we've got these solutions.

And it's different than if they bought it, they bought some product or something through a var, because they do have that direct tie back to the vendor. When you're that MSP in the middle, it kind of, it definitely clouds things up. Attorneys sometimes understand that, say they often don't. And at the same time, like in the case of this CIO, um, you know, I'm, I'm sure he or she would've been in the same boat say, what the hell are you talking about?

I'm paying for this stuff and you're telling me I can't even talk to the vendor. They're gonna be pretty, pretty p****d off. And so, you know, the, the, the vendor is, is it is not gonna, it's not gonna be a, a pretty situation overall. Um, and so, you know, we, we get into situations where we need certain information and, and the MSP is the one that has to initiate that ticket. They're the one that has to request those things because that's the way things work. Yeah.

And con consider this, right? So let's say Laura picks up the ir and I'm the cloud backup provider, and Laura and I know each other, and we want to talk, but we can't because our relationships are with you. We can't discuss your case unless you release us to discuss with each other without you present.

So that adds another layer of complexity that you're gonna need to start releasing your IR firms to talk to some of your solutions providers in order to maybe actually help with the IR and or the recovery. Yeah. And it also comes back to this coverage issue too. So we've seen a lot of cases where somebody said, yeah, we've reached out to, uh, Webroot and they've been involved, and Webroot Hass been doing all this stuff. Well, there's no attorney clients extended to, to Webroot whatsoever.

Um, and, and, and, and quite frankly, they're not approved by anybody to do the work. And now you've not only gotten them involved, which seems like from a technical perspective, a logical thing to do, uh, but you've also got them involved in, and, and they've also now, they're also now aware of the situation of your client. And so that, that can create its own complexities, right?

So, you know, one of the things that we try to stress very early on is you need to control that information flow and what is said into whom it's said into all those types of things. And it's like, you know, we get a lot of people, well, you know, we have this third party vendor that supports this system. We've already talked to them and said, well, what'd you tell 'em? Well, we told 'em, we got it with ransomware and blah, blah, blah, blah. We don't need to say those things.

And you can say, Hey, look, we have some kind of it issue. We're gonna need your help. Uh, we had something happen to our server. We may need your help to come in and, and help us get the the application back up. But you don't need to be so bad gum transparent that you're telling them all the, all the garbage, because a lot of stuff you may not exactly know.

And then who the hell knows who those people are gonna tell on the back end or who's gonna tweet about it or stick it on Reddit, or whatever the case may be. So you, you have to be very aware of these types of situations, environments, and, and again, in a co-managed situation, it becomes that much more, uh, tricky. Um, uh, it does not take too long. And Yvonne discovers that everyone on their call assumes that the backups, these are, the cloud backups are there and ready to be restored.

So it's obvious they've all been talking about that, that on the, that on the, uh, client side, they've been already telling the IR firm and the attorneys, yeah, we got good backups. We see the reports every day. We talk about it in our Q Bs with our MSP and everything's good. And then, uh, you Yvonne's the only one right now armed with the information that's gonna take a longer amount of time. Uh, so she has to explain the situation.

Obviously the CIO is not amused and immediately ask for a project plan to get the data restored, given what little time has passed since Yvonne spoke with the backup provider, she tries to sound as confident as she can, but the situation is not looking that good. Um, the CIO quotes, Hey, why we're, why weren't we ever told it would take this long? Uh, we pay you a lot of money every month, and this is unacceptable. So a little bit of what you would say word for word.

Yeah, word for word right there, right? We pay you a lot of money and this is unacceptable. And that's exactly the, the deal you got. So again, I was told, I thought you guys took care of all this. That's exactly, that's exactly right. Um, so people are obviously pod, um, but they have to figure out a different plan. And so now the conversation shifts to paying the ransom. And Yvonne, here's the amount of 12 and half million dollars.

So this is a big company, so the ransom demand is gonna be big. And so that's, now you got to say, holy crap, now we were doing these backups and we were doing good, but it's gonna take longer. And that difference between us being able to do it and then paying the ransom is now 12 and a half million bucks. And so, you know, um, so that kind of, that really starts to freak someone out, uh, especially someone out. And so obviously Yvonne needs help.

There's so many personalities and work streams going on here. So there's a few things I want to kind of, there's, there's, there's some reasons I put some things on there, the CIO asking for a project plan.

So really what you have to kind of be aware of, especially when you have individuals in leadership positions, especially IT leadership positions in a client, is that a lot of times these individuals have been around for a long time, and so they really feel that they know how to do everything.

And so that's a, a different, different dynamic when we go into a situation and there's not that internal, it, it's very easy for us to control what's going on, control the pace, make, make the decisions, and, and get stuff moving on. The problem with, you got a CIO and it could be a person internal, it could be a third party that they've hired as a, as a, as a fractional, CIO, whatever you want to call it. They will make things a lot of times more complex. They want stuff run.

They kind of, we're probably hired as a, a lot of these people may be hired because they're a project manager in a former life. They kind of think and act that way. They want everything kind of planned out. They want a thousand phone calls, they want phone calls every hour, they want updates and all that other kind of jazz. And so that, again, that's, that's important.

So it circles back to why we need to have these tabletops and why we need to have these conversations, especially with these key clients or these clients that have these a little bit more complex organizational structures, because we need to ferret this stuff out early on. You don't wanna be surprised by this, and you wanna know these personalities.

So you wanna know how to talk with these things and interface with these people and be able to, you know, appease them in whatever ways you're gonna do. You're not, a lot of times you're just gonna be mad the whole time, but most of the time these are intelligent people. And when the dust settles, they're, they're gonna, you know, for, for some of these things, you're gonna be fine. Now, on this backup situation, that's a whole different ball game to to, to deal with there.

And that's something you'll have to work with. Um, let's see here. Um, so this one from an MSP perspective took a lot of your MSP's times and resources. I mean, you had, you had to throw a lot of people at this. You had to, you had a lot of, um, you know, blood, sweat and equity into this thing. This was a major relationship for you. It's, it's, it's one of your key accounts. I mean, uh, without this account, you're gonna, you would, you would have to dismiss staff, you'd have to let people go.

Um, the renewal was coming up. And so a lot of these cases, it's amazing how this timing works. And so this isn't something where it's either they just signed this client or the renewal's coming up. It's never in the, the middle. And so that's where these things worked. Uh, the ransom was paid. You don't know the amount, you weren't really privy to that information, but now it was discovered that the threat actor came through a firewall vulnerability. And guess what?

That firewall is managed by you. However, Vaughn, the engineer on their side, has always insisted that he took care of the, he takes care of the upgrades, even though contractually that was your obligation, Vaughn wanted to do it.

So you let Vaughn do it, but there's no record of you having that conversation and explaining, okay, Vaughn, if that's your deal, that's fine, but here's the risk, and here, here's what we're gonna be responsible, and this is what you're gonna be responsible going forward. So, uh, in the end, there are signs that the insurance carrier may be looking for recoveries from you because they got the backup situation.

And now you got this firewall situation that from a contractual perspective, you are obligated to do regardless if V was doing the work. So, um, and so that this, this happens, the thing about the, and, and I wanna stress this firewall thing, and the reason why I used it on here is we're seeing many more times these firewall vulnerabilities. They're, they're letting them in.

So Wes and I were having a separate conversation the other day about this, but, you know, RDP and these simple ways of getting in were pretty prevalent. But now, you know, these firewall vulnerabilities, and you've seen the ones being, um, you know, notified out just recently with SonicWalls. I mean, Fortinets had their own, they all had 'em. Um, they're being exploited. And it's amazing how many times when we we go, look, it's not like the, the firewall was just one revision prior.

And, um, you know, people just lose sight of it. They're so focused on workstations and servers that they lose focus of these firewalls and they're so important to get done. Or, you know, the other thing you see is if, okay, maybe they weren't patched and the reason why the firewall, the vulnerability was exploited because that admin interface was open to the outside and you asked the service provider, well, why was that? Well, we need to manage the firewall.

And I'm like, yeah, you can manage it and get to the, get to the management interface. Well, you can lock that down, but you don't need to have it wide open to where anybody else can. They pull up that firewall, they see the admin interface. So those are the types of things that you gotta be all over it.

And I, you know, I've talked about this for years, you know, the definition of managing a firewall, um, the customer thinks that's a security type responsibility, and the MSP just thinks that's more of an IT type thing, keeping it up and running. And so when you get in these conversations and what that really means, um, it can get kind of tricky. Um, and then, um, you know, the data, there was data exfiltrated and the client is now have to start the reporting and notification efforts.

So what does that mean? So to kind of go back on the, the first scenario Yep, go ahead before You move on. Yep. So To say you ever any type of these relationships, um, and you, and you're supposed to manage that firewall, you probably should manage it no matter what the customer says. That's exactly right. Yeah, that's exactly right. Um, and if they're gonna manage it, then you better let that go and make sure you have some type of change order or something that reflects that, right? Yeah.

And it's probably 10 other things that fall into the same category. Look, before all this, even though I didn't think it was good business, you could go in and say, well, we're just gonna do this. We're just gonna do that and let the client kind of pick and choose. But this kind of changes, it changes your risk profile and you have to make some decisions. You're either in or you're out. And this scenario is really showing you why. That's exactly right.

And, and a lot of times if you just have the conversations with people and you kinda lay that out with the client, they'll understand, right? I mean, you, you know, you kind of put them in your shoes and those types of things and, and, and they'll understand it, right? I mean, you know, for you to let, like, for you to let go of one piece of the pie, um, is very dangerous in itself, right? 'cause you don't have the visibility or the control over that.

And maybe you have all these other pieces in place, but you're still gonna be the person on the firing range. I mean, in the line of fire. I mean, they're gonna go after you first, before they go after Vaughn at all. And believe me, Vaughn behind the scenes is doing everything Vaughn can to see why his A So, um, and, and that happens.

I mean, we've, I've dealt with situations where we've had people intentionally try to, um, to fire us because they know we're gonna uncover some stuff that, um, that makes them look bad from, from a role. And so you gotta, you gotta kind of be aware of those types of situations. So just like Gary said, you gotta be very well aware of these situations, and you gotta look at the risk from a number of viewpoints. It's just not one thing anymore.

Um, the other thing I was gonna say back on this kind of, um, notification piece, and back to this forensic data stuff, when they have to do, um, they have to do a notification and they don't know specifically what was exfil, but they know that there was some exfil done, um, then, then the notification becomes a very broad notification.

And so you can imagine the bigger the notification population becomes, the, uh, the risk or the probability of a a class action lawsuit being filed goes way back up. And those things take forever to deal with. So even if you were somehow able to save face in a situation like this, uh, that class action lawsuit that that CIO or whomever has to deal with for the next one to two years is gonna be that constant reminder of this event.

And you're gonna be, it's gonna be impossible to restore, restore that credibility. Um, you are, the other things I would look for is Here, and, uh, I, I tell MSPs this all the time, they love those big deals, right? They love those big deals better. Make sure that your company, your scale, your process, your discipline matches the responsibility you have as the environments get more complex.

And I know in the second MSP that I owned, you know, the owner early on stumbled over some big deals and was frustrated that I would tell 'em, we're not ready. Like, they have more expertise than us. Like you, you better be careful. It's not just revenue, it's responsibility, uh, with these environments. Chris, You're exactly right. I mean, I mean, let's say if, if you, if it's your first account that's this size, I mean, you obviously are gonna be amped up to, to grab it, right?

But you gotta be able to manage that account and be able to deal with that account, and you're not gonna be able to deal with it the same way you've dealt with all your other accounts. It's just not gonna be happening. I mean, there's a maturity level there. Uh, there's just a, it's just like managing the other relationship.

I mean, when you look at, um, when you look at other sales, you know, organizations, I mean, they have people especially equipped and trained to deal with these types of relationships different than they do with small businesses. And so you gotta kind of think about it the same way. You can't just say, Hey, I'm gonna go in with the same set of processes and procedures and reporting that I did for all my other smaller clients with these guys. I mean, it's just not gonna deal with it.

So we're not saying you're gonna turn down the big deal, but you better be able to kind of get on the table and deal with those types of things. I mean, I mean, heck, I remember, uh, or You might turn it down, Chris, Say that again. Or you might turn it down. You might, yeah. You, yeah. You may not get you, once you start putting all those numbers together, you might find out that it's not worth the money that you're making. You're exactly right.

Um, and so, you know, these guys, um, typically, especially if this is a prospect that is outsourcing for the first time, they probably don't know as much about that model as they should. And so they are looking to save money. A lot of times when they're trying to outsource you, they're trying to solve a problem with, they can't get resources or they're trying to reduce expenses. And what they don't understand is probably not gonna reduce their expense whatsoever.

But if you come in there at a low dollar amount, there's you, you better be thinking to yourself, um, and they jump all over it. There's, there's a reason for it. And so, um, it's, um, it's, it's, it's a different ball game. Um, and you just gotta, you gotta be aware of it and you just gotta be on top of it. I mean, even the tool set might be different.

You might be using one antivirus product for these clients and this client, you're gonna step up and use something different, um, because of just what they are, the complexity, they're in the industry and you know, like you see MSPs, uh, maybe they've never, never had the opportunity to, uh, have a, a law firm as a client and they just go in and think it's the same. No way it's the same, right?

And so you better reach out to a peer or someone else that you know, that's maybe not a competitor with you and ask those types of questions and what they're getting into, uh, because that's a whole different ball game. I mean, when you're dealing with a law firm, um, you know, for them suing you and dealing with that type of stuff is a, is a, is a low cost thing. Um, and a lot of times those law firms are gonna do everything to avoid any type of blame or anything of that nature.

I'm just kinda speaking in general. They're not all that way. And so you just gotta be prepared for, to play that game. Um, and you better charge for it accordingly. I mean, um, I have a, a friend of mine who years ago sold a phone system, uh, to a law firm and they've set it up in the three payments and the law firm, uh, paid one and said, come and sue me for the other two. And so, um, it's just, you just gotta be prepared for those types of things. Um, so that was really that, that scenario.

And, um, again, I pulled in things from different ones, but I can tell you that dealing with internal IT and MSPs, and sometimes we even deal with multiple MSPs when they're spread out geographically. I mean, we've dealt with situations where we've seen that the, um, that the, the, the MS P has some other relationship in addition to the MSP with that company. They're a brother, an uncle or cousin or something of that nature.

And so you gotta take those kind of things into account and they can make things, uh, really difficult. So we'll go some for some review. 'cause I did wanna leave plenty of time for discussion because I know we usually always run outta time for stuff. Um, we just talked about it. You can't, you gotta risk assess your clients. And I should also put in there, you gotta risk assess those. It's gotta be part of your process, um, when you're going through these deals.

And, um, if, especially if there's some material change, it's funny, in the banking world, if you had some material change in the environment you had, that triggered some types of events that had to take place in your bank. So let's just take it from a vulnerability assessment perspective. We're in the banking world, you did monthly assessments, that's fine.

But if something changed in the meantime between those two assessments material that triggered the need to do another vulnerability assessment, same thing with a pen test on an application. So the same thing from your perspective, if a client is somewhat, you know, materially larger than what you've dealt with in a different industry, you've dealt with, maybe there's a different organizational makeup.

The fact that they have a location overseas, or they have a location maybe even in a different state, and you've never handled that before, you need to kind of deal with that and sit through and risk assess that in the banking world. We, you have to risk assess every living thing in the world, right?

That you, you gotta do it, you gotta demonstrate it, you gotta document it From an MSP perspective, as important as you are, and as important those clients are to you, you need to do the same thing. And just like Gary said, you might come down to that decision and say, Hey, we're gonna pass on you because it's just, it's just not a good fit.

You know, you don't have to say you're, you don't have say, Hey, you're too much of a high risk to us, but you can just say, Hey, the, the, the fit's not right and we're gonna walk away. Um, Not to mention Chris, when you're in this situation and, and you're trying to make good decisions, when, you know, if you lose that client and they're 15 or 20% of your recurring revenue, it changes how you think about things, right? Like I always tell say that every big wave hits shore.

And, uh, so you can't even approach it the same way to, to be pure in your decisions. 'cause you're thinking about your own business and the impact That's, that's a, that that's a good point too. So you could look at the, the, the, obviously the, the immediate risk, but then if you do lose that client renewal, whether that's a 12 month or 36 month, or even in between because you know they've let you go, yeah, you need to think about the ramifications of that.

You gotta think about, Hey, you know, I had to hire people for this relationship. I'm now gonna have to fire people for the relationship. I had to hire people and I had to, I had to lease more office space, and now what am I gonna do about that? There's just a bunch of factors, you're right. It's a, it's a, it's a business decision.

Uh, just like you try to convince your clients that to take cyber risk as a part of their business risk decisioning, you need to be thinking those same things in, in, in the, in, in the same vein here with, um, with, with regards to bringing clients on. Um, so you must be aware how your response may vary from client to client. What I'm talking about here is your incident response.

If you think you're gonna have one incident response plan that's gonna cover everybody, um, you, you're wrong, right? So, and that's why I created the different scenarios here. Um, you may have, you may have a good template and a good baseline and all those things from this response plan, but you need to have some, some different paths, some different options and maybe even different ways of handling a situation, uh, with one type of client versus another.

It's easy to, to think about a regulated client versus a non-regulated a client, um, and how those types of things. But we've even had non-regulated clients, which you just on the surface peer non-regulated, but their customers are regulated. And so that adds a different, different complexity.

We've had issues where, from a legal perspective, there was, there was no, there really was nothing that the, the, uh, victim had to do, but their contract language with their customer, with their customers said there were things they had to do from a reporting and notification perspective. And so that's why you, you have to be kind of, um, you gotta ask those questions first. You got to get to know your client, you need to do the tabletop exercises. So those things get farted out.

And then you can have, you know, a, a robust response plan. Is it gonna be a thousand pages? Absolutely not. Um, are you gonna have to write a separate response plan for every client? I hope not. But you just need to be able to know that you're gonna have to tweak flex and that type of thing.

I mean, you may find out that, um, if we just want to use one of these guys in his example, Brad might be a great fit in an incident response situation with 50% of your clients, but the other 50%, he wouldn't be a, a good fit just with his personality or whatever. So that's going, and I can't understate this enough, exfiltration has changed everything with regard to everything. Um, I mean, I just can't put in that enough. I mean, the, the threat actors know that.

Um, it gets really tricky, um, especially when you're dealing with SMBs and you have owners that have their own personal stuff that they keep on their company's stuff. And so we've been in, and so like, uh, it's weird when you have an owner that says, look what my company does. I really, if that gets published, there's no harm.

But there's some personal information I have out there, some personal financial information, and you don't have to go into detail about that, but there's personal information out there that if that got published, man, that would really be bad. And so that really gets, it's so emails.

He doesn't want his wife to see Emails you want, you know, it's funny, when I worked at the bank, there were many times where I say many times, but there were times where a teller would accidentally reveal to a spouse information about a bank account that that spouse wasn't part of. Um, and, and that was the, that was the, the booty bank account. So, uh, anyway, uh, you're exactly right, Gary.

So, um, you know, there's stuff out there and there's stuff that, there's information out there that maybe from a legal perspective doesn't fall under any definition, but, um, if it was made public, it could embarrass just embarrass people or, or just do things that, that just aren't right. So it does change everything You need to be having conversations about their data, what's stored there.

Um, I know of a case where they had a, their attorney and their attorney stored stuff on their servers because their servers were secure and all this kind of good stuff. Well, when they got popped, obviously the attorney's data was now under the scope as well, and like, holy, you know, and so he's shaking his head as well.

So, you know, um, we, we have a situation, um, um, where you have, um, two different companies owned by the same people, but they're two different companies and they share servers and it's the same IT provider. That's ugly. Man. That gets really weird.

I mean, we, one of the, I remember one of the first things I ever had to deal with was that situation where it was a company who had a, like a, a brother or uncle or, or somebody like that, the owner that was in an office in their same place and he used their servers and stuff and he got hit with the ransomware and it spread to them. He didn't have insurance and they did. And it was all this kind of crazy back and forth and trying to figure out who's gonna pay for what and nasty.

So that's another thing is if you've got these types of arrangements that are not clear and cut from a, who's got what and who's using what you need to like figure that mess out, you need to separate that stuff. You don't want companies mixing, mingling data and the same stuff, just like you have multi-tenancy in your tools. You really shouldn't have any clients kind of, of mixing and matching stuff. And you should have those conversations, especially around that personal stuff.

You should have those conversations and, and figure out a different solution to keep that personal stuff away from the company stuff. I never thought about that, Chris, but I, I know that I've had many customers, maybe not couple scenarios where they were two different people that knew or related to shared stuff, but a lot of times a company might have two or three entities.

They have a retail piece, they have a distribution, but they're different corporations with different liabilities and different customers. Um, but they share technology. And so that's pretty common. That's one that everyone needs to think about. 'cause even if it's the same owners in both, they still wanna split and limit that liability. That's exactly right. And if you don't know that that stuff's going on, that's, that's a different issue in itself, right?

So, you know, I don't know how many people would know or not know that, you know, an attorney was storing stuff there or whatever the case may be. But, um, you know, that's, that's where you gotta really know your client and have those conversations about data security. I don't, I mean, I don't wanna go down that path and, um, I think there's been, you know, cyber calls in the past around that.

But, you know, this concept of better securing data and uh, encrypting data at rest, um, goes a long way. 'cause it helps in the exfiltration side of things that the bad guys exfiltrate data that's encrypted by you, then they don't have any leverage with it, right? Um, they may have a little bit with file name and stuff of that nature, but, but if they don't have access to the meat, what's in the actual data, it goes a long way.

And so I could tell you I've had tons of conversations with people and it's, it's gone very well where they understand that there are, there are very, um, sensitive portions of their data that they need to take additional steps to, to secure. And now is the time to do it.

I was on a call early this morning about somebody that is, she's going through right now, her file server and reorganizing everything as painful as that year is going years back, uh, reorganizing everything, uh, to get prepared to do thing, to do these things like this. So it has to be done, and you need to be having these conversations and that goes into, they, your clients are gonna have to do more. You're gonna have to step it up every time there's an iteration.

Uh, I guess what you would say is there's an iteration, uh, with your clients all the time. There's a new iteration of stuff that needs to be done and, um, you know, some of it you may be able to, uh, absorb, uh, if you've been able to, you know, reduce, uh, reduce your, your si, you know, you, you, you basically increase your margin with the client and you feel that there is some, some room in there that you can add some more value to.

Uh, I'm sure you can do it that way, but at the same time, uh, getting this message across from a security perspective that they need to do these things and why they need to do these things. Uh, I used the term quadrillion yesterday though. There's a quad drilling examples out there of why they need to do it. So it's, they can no longer use the excuse. It's not gonna happen to me.

One thing I was gonna mention earlier is like if you're in a major metropolitan area, the likelihood of a client getting popped and hitting the news is much higher than obviously when you're in the country. And so you can kind of take that into account in your stuff. But my point is, is all those clients have some need to improve security at some point. And just like you, you know, the best of EM MSPs out there are doing budgeting and planning and stuff every year with their clients.

You need to be budgeting and planning for these security things, uh, and getting them done. And then I can't underestimate the legal side of things. Um, you know, Wes was talking earlier about his examples of doing the tabletops of the banks and getting the players involved. And, um, I I stated this in other sessions is it doesn't hurt if your client is willing to do it.

And, and, and maybe even, I mean, obviously willing to pay for it is in those tabletop exercise, have a breach attorney involved in those because the breach attorney is an active participant in these, these efforts. And so the, the larger companies have already been doing that, meaning they do tabletop exercises that includes the technical legal and those types of things involved.

So if you have clients, especially like these ones we're talking about with these maybe that are a higher risk, um, bringing in a breach counsel, uh, an attorney in to participate in those things will go a long way. And I think it, again, it shows, um, it reflects very positively on you that you're thinking about those things for your client and you're bringing those resources in. You know, Chris, I hope what people heard a couple things today.

One, you can't just have a technical relationship with a customer anymore. You have to understand their business. You have to understand their customer's business to some degree, uh, or you can't keep 'em secure and you can't be prepared, uh, when, when there's, uh, an incident.

Second thing is based on everything you've talked about over, over this time period, there's no scenario where you start to, uh, between this and what we talked about with Wes yesterday, where you start to have a different conversation with the customer about what needs to be done, and they're really not gonna put up a huge fight about investing more, right? Right.

And also translate this like we're gonna do in a little bit on the next session about questions we ask pro uh, prospects, uh, to, to, to create a, a, a wedge. So, um, one, you don't have a choice, you need to do these things. And then two, uh, it's it's also good business. You know? That's exactly right. It's about having those business conversations. And I, and I think he's on here, I'll just call him out.

I mean, I mean, Sonny Lowe, you know, personally is one of the, the best guys I've ever known to be able to have those types of conversations, or at least, you know, coach people and have, have those conversations. I mean, that's his approach. And if you've ever met him or you ever talked to him or listened to him, said anything, he's Awesome. He's awesome, and he, he knows how to get it done.

And, and, you know, for, for people that are really kind of technical from the minds, you know, from the, from the get go and then, you know, you know, start their own Ms P and do those things, you know, those are, those are tough hurdles, but you really need to push that envelope or find somebody that, that you can hire, that can have those types of business type conversations with it.

'cause if you can frame this stuff in business terms, that goes a ginormous, uh, way to, to explain these types of things. If everything's just about new gadgets and gizmos and that type of thing, I mean, that's a, that's a tough hill.

Um, but you talk about things and then again, like I talked about on the IR tabletop, if you have a, a breach attorney come in and that breach attorney is, is basically backing up the stuff that you've been saying for months, and they're, they're coming in basically saying the same things, it's gonna resonate even, even even more with your client. All right. Do we have some questions out there?

Well, I, I've been at, I've been seeing if anybody will pop 'em in Chris, but it doesn't look, I'm surprised. Well, maybe, Maybe because you're not showing your face is maybe why they're scared to ask for. I know. I've been asked, I thought it would promote more questions. Um, but you guys did a fantastic job as always. Um, Gary, I think, you know, it's interesting, uh, is Sunny still with, you know, I mean he's been with True Methods, I think for a long time he's was in peer with you guys.

And you know, it's a testament to, I think, to what you guys are doing and what you're teaching, Gary. 'cause that's a, a big part of, of, of, of your peer and, and, and building a sales engine, building wedges, et cetera. Yeah. And, and so much of it, Andrew, is the same things you do to change your relationship with your customer, right? Those are the same characteristics that allow you, um, to be more successful in adding new customers at the right price.

So, uh, I always say, uh, in this business, sales and VCIO are really very close to the same job, just ones with a customer and ones with someone who's not almost not a, you know, not a customer yet. Yeah, there, we got a question, but the last thing I'll say about Sonny is, again, another guy that walks his walk. I, I, he posted something up on LinkedIn, I think it was probably four or five months ago, about his, you know, completing SOC two.

Again, the guy's doing the guys, the, the guys and gals selling the most at the highest price, doing the best. It's, it's a tremendous commitment. And they're, they're doing all the hard work. All right, so question here for you, You always try to balance it. Andrew, today was a lot about the downside and the fun and the risk that we and our customers have, but we're always trying to balance that with the opportunity side.

Uh, the fact that this is, this is good business and, and the best MSPs have to be good at this. Yeah, the story Doubt. Alright, so, um, question we got, uh, what's the best way to handle the conversation with SMBs? That they have a responsibility to know the legal requirements, the legal requirements they're under. Under. Yeah. So I will, I'll speak first on this one. And where I've had success is kind of, um, comparing it to what the same thing they do with an accountant.

Understand their financial responsibilities and their tax responsibilities and those types of things. And so it goes back to what we just talked about, about those business conversations and framing 'em in the sense of business. It is a, it is a business risk and is a business need. And that legal exposure is very key. We didn't talk about today, uh, as much, but, you know, ransom payments, uh, you go off and you make your own ransom payment.

Go buy your b your own Bitcoin that you just opened yourself up to a ton of legal exposure. So the, the, the, you know, the same thing goes on for, for making decisions not to do things or to do things a certain way or to not bring an attorney in or whatever those things are. You need to have those conversations. And at the same time, do you need to have all the answers? No. But if the, if they come to you and have that conversation, great.

And if they need more answers, at least you can say, well, let me reach out to a, a, a breach attorney and bring them in for, you know, 15, 30 minute conversation to talk about this thing. So, um, that's, that's my spin on it. So, uh, I have a few thoughts on some of this too. Like what I would do in a situ, I, he's Chris is right, it's business risk, right? And so use, use the topic of insurance as your, um, excuse to have the conversation.

So what I would do, I would sit down with a client and say, Hey, uh, Andrew, you're my client. Um, you know, there's some new things that have happened inside of cyber insurance coverage that we need to have just a business risk conversation, right? And they'll say, okay, that, that's interesting. Tell me more and say, you know, Hey, I know you've got cyber insurance, it's really good. We've helped you look at it. Or if you haven't, you might wanna have that conversation too.

But you say, um, that, that's really good that you've got it, we actually have it too. Let's talk about an incident. If we ever have some kind of security incident, we need to make sure that we both understand the requirements here. Because when it might be a security incident, yeah, when you have a security incident, we need to talk about this because, you know, you might want for me, Andrew, to just pay the ransom and move on.

But I may actually be completely unable to do that because of how the incident may occur and how it may happen. It may be that my, in my insurance will simply say, I cannot and will not pay for this cyber, uh, for this ransom. You are going to have to pay for it. Just, just like to shock them into thinking. Now again, ideally I'm not advocating paying ransom, right? But use that as a starter to get them thinking of like, oh, okay.

And then say you have some responsibilities that you need to be doing that you are required to be thinking about. That due diligence are, are part of your, your requirements here. I know that I handle it for you, but when it comes to security itself, and it comes to how this is gonna be handled between your insurance and mine, there's some differences and you need to be prepared for that conversation of the differences in what's in it.

But that's how I would open the conversation because that gets them thinking. It lets them think we're talking business risk. I shock them into thinking why they need to have the conversation. Um, that that would be some of my recommendation. That's a really good question. Yeah. And on that ransom one, that's a great example, right? Because most policies are a reimbursement policy when it comes to extortion payments.

So if the demand is a million bucks, the client has to come up with a million bucks, they're gonna get reimbursed for the million bucks, but they gotta have a million bucks. And if they don't have that at, they don't have that at their disposal, they're not gonna be able to pay it. So that's a good example.

Well, one, and you've said this multiple times and you said it yesterday and you've said it before on the cyber call, is, do you, do you even know, uh, if all your, have you had a conversation with all your clients even about their cyber insurance policy? So we're, we're good Because we've been surveying people, Right? Yeah. And like I said, you better keep track of that, right? You need to know when those renewals are and who the carriers are, because that will help you immensely, Andrew.

And not to mention you heard what that conversation that West just shared, having that conversation is part of changing your relationship. It is part of having them see cybersecurity as a journey explaining what happens in an event. Uh, couple that with, you know, explaining the, from a high level, not a technical level, a framework, uh, like, like we did yet on, on, uh, the call about cyber resilience. All of this both in sales and VCIO is how we change those relationships.

So that was perfect, Wes, It it, well it, Gary, if I could also say we're, we're seeing is we're going from checkbox renewals to five, six, and seven pages. It's a perfect time to a be setting that expectation with the client that, hey, we're gonna have to be involved. Let me explain to you what the, the renewals look like these days with a carrier, number one.

And then the other thing is, it's a perfect time to, you know, everybody's thinking about your contract renewals when you increase prices. Maybe that's the perfect time to start talking about, Hey, look, by the way, last time, you know, your customer didn't ask for an internal and external application scan last time, the cus you know, they didn't ask about this, this, and this. Don't just take my word for it. Look at what the insurance carrier's asking for, right? Yep.

So, Absolutely, uh, and once you do this, I can tell you, um, price is, will not be the same objection. If you're feeling that today with customers and prospects, you will not view that or, or have that as an objection in the same kind of way guaranteed. Yeah. Yeah. Matt Flores has a good question real quick about what do you, what do you do if you have, you know, smaller clients that kind of push back on purchasing cyber insurance?

I mean, again, you gotta have that con and and again, you need to relate, hey, I as an owner have cyber insurance and this is why I have it. And you gotta put those in the same terms. And a lot of people think they're insurance poor in the scheme of things. When you look at insurance rates, typically, uh, cyber is very affordable still. And so they, maybe they haven't explored it. Uh, and you may find out that their broker just hasn't done a good job of, of explaining it either.

So you don't wanna try to sell them a policy. I mean, there's a little bit of licensing and legalities around that, you know, usher towards them. And if they don't have a broker that's doing a very good job of explaining the benefits to that, maybe that's the best time to introduce 'em to a broker that does. Chris, you can't, you can't have a mortgage on a house without homeowner's insurance. That's exactly right.

Register a car without car insurance and we're, you know, there's no law, but we should be to the same point where you can't be our customer if you don't have cyber insurance. That's right. Yeah. Matt, we've had a guy named Justin Rhon, we'll have him back again on the cyber call. Gary's gonna be doing, we're, we're building some security tracks, right? Within true methods.

You know, it's, it's really like Gary and, and Chris are saying right now, it's gonna be one of those things where you're gonna have to think about, is this somebody I can actually keep as a customer? Because, you know, what does that do to your risk if they're unwilling to do something to pro to provide that coverage? Um, and so, so we'll get more into that. Uh, fair, fair enough, Gary? I I know we're gonna be addressing that. Yeah. Um, that, Ryan, you started to answer a question.

Um, maybe you can, can we get your perspective on what Matthew was asking? You know, what are good examples of a situation that calls for incident response? I think what he's saying is what transitions right from, um, uh, uh, something that you know is is, um, an event to an incident? Is that, is that a, is that how you took that? Yeah, it is.

I mean, I think there's this distinction between a security event and then an analysis of that event and the determination that that event is an incident or what we call declaring an incident. And so really what you're asking is what is the criteria for which to declare an incident?

And, and the fuzzy answer I gave is any situation where you think there's been inappropriate access modification or disclosure of, uh, of data that might especially cause damage to you or, uh, you know, a loss potential to you a customer or have some sort of regulatory requirement, um, around that event. And, and so that could be very broad, like a, a phish, like let's take a phishing email, right? So you get a report of a phishing email. Is a phishing email an incident?

Well, I don't have enough information yet. Did the user open the link? Did they open the file? Is the file actually malicious? Is the URL actually malicious? Oh it was, okay, I have an incident now I have to respond to this. 'cause it could lead to inappropriate access modification or disclosure of data. If it's just an email and no one's opened it yet, delete the email, you're not in an incident you've successfully responded to and protected against that event.

So it's, you know, you gotta, you gotta really think about what your triggers are for declaring an incident. And I'm sure Wes has some more concrete thoughts around that. Well, I do have some thoughts. I'm gonna paste the link in here. Um, make sure that's the right one. It is. So check that out. This answer is a little bit of Matthew's question and up above Keenan's question as well is like, how do you go through all this?

And, and there's a little bit of art, there's a little bit of science, right? Just like, uh, Ryan was mentioning there, there takes some time, some, some research into, okay, I have some kind of event that's occurred, what do I do with it? And when you're going through this incident response process, like you can, I, I think a PSA is a great system or could be a great system to use for your incident response tracking, but like, just let's keep it simple right now.

It could be sim as simple as a spreadsheet and that spreadsheet just lists the incident that has occurred. Um, any associated information that you have with it, it could relate to tickets or whatever. But if you look through that article I showed, it's really nice 'cause it gives you some things that you can think about as you're kind of researching this. So like the second phase of identification, you know, ask these questions of when did this event happen? How is it discovered?

Who discovered it? Have any other areas been impacted? What's the scope? Those kinds of, there's more questions, but those are some of the great questions that you should be asking and documenting inside of that incident to understand what's the breadth, scope, severity, criticality, all of those things. And that should be a very rigorous documented process that you go through that should then output into some amount of criticality. Um, this could be an all hands on deck criticality.

You pull the plug, you know, the network plug so to speak. Um, this could be something that you just track and you maintain and you work through. It could be something in between. Um, the containment phase comes next and I love the questions they've asked. What's been done to contain it, uh, short term and long term. Has any malware been discovered? I'm not gonna read all these, you can see it, but that article I just posted is a really good example for you to build your IR processes.

Now, your IR playbook or the overall program that Mike Briga and Chris Laer shared yesterday, that's how you build it. But the next thing that comes into place is what are those procedures and things in place that you have to make sure you document this process all the way across the finish line to where ultimately you have the whole thing eradicated, you have the thing recovered and contained, and then you do get into the lessons learned of what you did about it. That's how the process works.

Um, it's not difficult and it can be as simple as an Excel spreadsheet, um, but uh, that article will give you some help in how to communicate that, how the client should be involved with it. All that. Chris, any thoughts? Yeah, also, you just gotta be sensitive to who it is, right? So I mean, you have, um, healthcare type, uh, organizations that may be more sensitive and want more of an investigation done, even if it's maybe looks, feels smells, uh, not critical, but they still may.

So that mean you gotta call insurance, but they may be looking for more of an effort to just make sure and rule that thing out ver versus another client. They may not be that interested in doing so. So you're gonna kind of know your expectations of your client on those types of things. I can tell you that, you know, we get, um, people that have, let's just say recently been bit, uh, they're super sensitive to everything that goes on.

And so you just, you need to be able to show that you're kind of empathetic to that and, and deal with those types of situations. So, um, what everybody said here is, is correct, but again, you have to kind of tweak it and press different buttons for different types of clients. Yeah, I would, I would encourage you to think of your IR plan almost like a state machine, right? What state am I currently in? Am I in the identification state?

Okay, what are the conditions that are gonna need to occur for me to transition to another state, right? And like, you know, like one of the things you can get in is like you can get an event in and like one of your line techs could work on that for a week and never escalate it and never tell you about it.

And there should be a trigger in there to say, if within a certain amount of time of triaging a security event you have not yet made a determination, you should automatically transition that to another state. And that state change should come with an additional set of procedures. Like for example, state change for me, the moment we declare an incident, my legal counsel, me, my PR team, the engineering lead for that area and my entire IR team getting a, getting a communication channel.

And that's where we start communicating about the event. Everything is contained in that channel and that's how we start working our way through the plan. So think about it like a state machine and think about how you're gonna transition between those states and how long you wanna operate in a given state, which may be subject to like, you know, like Chris said, healthcare, you might have very strict reporting requirements within a certain amount of time.

So your state machine for a healthcare company is gonna be much more condensed than it might be for another entity where you don't have such strict reporting requirements. Yeah, it kind of reminds me of a DJ where you kinda may have to speed up the beats per minute for one song worth another song and do it, right? Yeah. So same track, a little bit of a, So that I, let's put a kind of a bow on it right there. Um, man, that was just fantastic.

Jason Slagel, literally, uh, as you were saying, state machine Ryan, he was typing it. So looking forward to your session coming up Jason. So Gary, kinda let's take a, a step back and, and, and I'm gonna thank these gentlemen in in one minute, but where we've come from, you know, Ryan and Wes kicked it off yesterday with, you know, looking at building cyber resilience, operating in an assumed breach.

Then we had Chris and Mike, uh, walk through building an incident response plan today, running it in a sense through a tabletop, all of these things, you know, again, going back three, four years ago, Gary, we're never even part of a conversation running an MSP, but let's just take it, start there, right?

Um, and, and so, you know, we're gonna bring you on with Mike regard, Cecil Marco, Matt Solomon from ID agent, um, and look at go to market because again, yes, we have to be doing this internally. Yes, we have to set up all these policies, but we've gotta have the rubber hit the road and we've gotta make some sales outta this. So kind of give us, set, set the stage here first, Gary.

Um, When we say go to market, I, I mean taking everything and, and now being able to translate this both customers and prospects. 'cause you have to be able to do it with both. We have to change the relationship with customers and we gotta get new customers and we gotta get people to easily pay us the right price because you mentioned there's so many things that we weren't doing a year or two or three.

And what that means is to do all those things, um, you know, you have to be able to command the right value and it's not a price increase, it's a value change. And what we're gonna talk about is explaining to customers and prospects that their costs have already changed. Yeah, they've already changed. We're just explaining them to it now and we're giving them a more efficient, uh, way of controlling their risk and, and having those costs be more predictable. Yeah. Fantastic.

So back to Wes Spencer, Chris Laer, um, always guys, just a fan. Fantastic man. I love, love, love your tabletops. They're always so heavily attended. Chris, um, you know, your wisdom of being front and center in these cases. Um, just, just, just, awesome. Thank you guys. Um, so let's just talk about the transition from in the next four minutes. Um, I'm gonna end this session. We don't want you to go away. So this session will end.

We will be quote unquote down for a minute or two and I'm gonna pull the way crowd crash works is you end a session, it'll start rendering it so you can watch it again. And then I'm gonna pull you all into the next session. Um, we'll be behind the green room for a minute, getting ready again, don't go anywhere. Um, but, uh, any closing comments, Gary, before we close this one out? Just thank you so much for everybody in there and, uh, attended today and also, uh, thank our, our team. Yeah.

All right, well again, thanks everybody. We will be right back. Give us just a few minutes and uh, uh, we we're so glad you're here with us. Take care.