How an MSPs IR Engagement Dramatically Changed Their Sales Conversations

Okay. All right. Welcome everyone, and happy Monday. Happy St. Patrick's Day. Um, any, any of y'all celebrating out there? Um, I don't know, Chris, you uh, you doing anything there in San Antonio? Does it get wild? It get wild on the weekend? Uh, so wild that, uh, the always add a little glory. There was a, uh, shooting, there's a restaurant down if anybody's been to the, down to the Riverwalk Casa Rio. Uh, one employee walked up to another, just shot 'em in the head. So that's St.

Patrick's Day in San Antonio for you. So that's what happened. Wow. So I think about that. You lose two employees at, uh, one time in that case, and no one knows the reason why as of this thing. Mm-hmm. But, but yeah, everybody did their celebrate, I think on Saturday. I don't know much going on today. On Monday. Yeah. Andrew, I used to for years.

Uh, you know, my son and I with different groups of people, we always went to the Philly Spring training in Clearwater, and we always happened to be there for St. Patrick's Day. And Clearwater's a fun place to be for St. Patrick's Day. It's, it's actually pretty big here in Tampa area, Gary. I was surprised. They, they do this whole big thing where the Hillsborough River, they turn it green and, and, uh, but, um, interesting. Um, well, and I remember Ours, city's always green.

I remember meeting you and Gary, I think one year at, in Clearwater. Yeah. Uh, for Phil's game. So, uh, yeah. How about, how about you, uh, Eric, any, uh, any interesting things going on in Cali? No, I was surprised. I was in San Diego yesterday and I couldn't believe how packed the bars were. It was just wild. Interesting. Interesting. Well, happy Monday to you all out there. And then Bob, lastly you Lafayette, any exciting things going on there?

Oh, look, Anytime there's an official reason to drink and eat too much, we're all in down this part of the world. So Yeah, of course. That, that is definitely a fact. Yeah. Yeah. Uh, alright. And I think there's a lot of spring break going on right now in terms of whether, if you have a high school kid or, you know, co sorry, kids in, you know, anywhere from grade to high school and even collegiate. There's a lot of that going on right now, but let's get on into it here, um, on the cyber call.

So, um, I, I really enjoyed last week, um, Gary. I thought that was, um, you know, having Clark on and, and going through, uh, the threat report. He's, he's always fantastic to have on, and I can't believe it. We're only probably about six weeks away from reaching out. Um, I'm gonna do it earlier this year to Verizon. Um, this will be, I think probably our fourth year of going through the Verizon DBIR.

And that's always a fantastic, um, threat brief and, and prob obviously the, probably the gold standard. Right. Chris, would you say in terms of threat reports out there? Oh yeah, definitely. Definitely. We'll Be going Andrew. I'd wanna know, like with these kind of like, you know, well-respected reports, how many MSPs are using it, or a summary of them, like in their conversations with customers. It's, it's great. Hey, I'm just, well, we have two right here, Bob.

Have you ever leveraged it in any way, shape, or form? Oh yeah, absolutely. Especially, um, because not only internally, but we also turn on a lot of, um, our clients Yeah. That information as well. Right. Because it's not something they'd typically go out and seek for themselves, so, yeah. Hundred And you have some bigger clients too, Bob, so you can go based on vertical or what's going on with threat. That's right. Threat actor for that vertical. That's correct. Yeah.

Government, government, medical, energy sector, those are all people who really should be the most concerned. So we do, we actually have, we inform them on it and do a briefing. That's cool. That's cool. How about you, Eric? Any, have you re used any of the threat reports on your side? We, We reference it quite often, but not by name. So we'll pull up what those, the threat actors are and the methods and tactics. Right, right. How about you all out there? Any of you?

Um, Ann, welcome and happy St. Patrick's. Did. How about you all? Um, any of you used some of the threat reports and things we've done over the years here on the cyber call? I'd love to know. Um, so, so in in regards, uh, of, of last week, you know, it was interesting, um, one of the, uh, stats was this, you know, immense reduction, uh, 79%, or I shouldn't say reduction, but 79% of attacks listed in the CrowdStrike report were malware free.

Um, Chris, you have a bit of a different, different take in your caseload. Um, what did you say the number of ransomware? I think we're working about 60 with six zero right now, is what the number I heard last. Yeah, yeah, yeah, yeah.

Um, what do, what do you think kind of maybe where some of this discrepancy, or I'm calling it discrepancy, is, is it, you know, you know, the types of, you know, actors that maybe CrowdStrike's more, you know, focus more mid, upper mid market enterprise than what you know, you may be dealing with, or just the, the ones that are bubbling up to you within Solace and CFC we're not, are not, you're not getting the ECS and the, you know, more air coating benign, like what Yeah, What I described was ransomware cases.

I mean the BEC side, they probably had, they have just as many, I mean, I think a lot of it has to do with number one is it's a tough thing to, to grab. I mean, CrowdStrike, they deal with cases of all shapes and sizes. They, you know, they had some bigger ones, right? I mean, you know, a two, 200,000 endpoint case is not out of the norm for them, where we wouldn't touch anything like that. And I just think a lot of it has to do too with just the, the identification of these groups.

There's, they're not these like handful of groups doing all the, the attacking. It's more spread out and there's just little groups. I mean, we've had, we've had groups that just focus on a specific industry, like maybe like, uh, cosmetic surgeons or something like that. So I think it's just, uh, it's just a, I don't wanna call it a weird time. It's just a different time with respects to just the identities of these groups and attacking.

And it's not like there's some central database where everybody's saying, Hey, I'm checking in. I got attack today. Yeah, it's, and just curious, since we do have your care before we get into today, what are you seeing in terms of MSPs? Because you know, you've obviously over the last, since you and I have been working together, gosh, now for nine years or so, we had that peak obviously with MSPs and rms, or do you still see them being attacked and, and what are you seeing?

Yeah, so we, we, I would say especially in the last few months, we've seen more MSPs attack, but it doesn't, like they were targeted. It looks like they were just part of a broader approach. And then somebody got in there and figured out they're an MS P and just started moving laterally into their tool set and doing what they do.

I mean, we've had cases where, uh, you know, and does seem like either the, they came through the MSP or they leveraged credentials collected by the MSP collected from the M ms P somehow some way, um, and then use those credentials to get into their clients as well. So yes, we are seeing that, but again, not seeing anything targeted.

And then as far as MSPs being involved, it's, it's, it's, it's, um, it's, it's kind of, we've seen some MSPs out there that are, um, how would you say tr maybe trying to disguise the fact that something went wrong from their clients. We'll just leave it at that. Uh, and then their clients kind of find out through the threat actor communicating directly with them says, Hey, we've been trying to communicate with you. We have your data, but you haven't made contact.

And then the client's like, oh, this is the stuff the MSP told me to ignore that it's just spam or some scam, or something like that. So we've seen that, we've seen some MSPs that get in there and try to pretend they can do ir. And that's usually, um, not a good thing. Um, Well, they're not pretending they think they can. Well, yeah, they, they think they can. Yeah. But in the end, they're pretending they just don't know they're pretending, uh, or, you know, their denial.

And then we, we, and, and so yeah. So that's, that's what's going on out there today. It's, uh, and I, and I mentioned this too, I'll mention this on this call. We, we've seen, and I get it, I've talked to some MSPs out there, there's some MSPs that are getting very territorial 'cause they're afraid that, you know, someone like us might move in and try to convince their client to do security business with them, or the, or the MSPs providing them security tools.

And so, you know, um, so that happens too. So we see, Well, based on that, it won't take much convincing. Yeah, I know. I know, right. But they're, uh, yeah. But we've had some MSPs just refuse to let us install tools or gain access to their system. Yeah. And somehow, yeah. So it's crazy. They throw up the, you know, they throw up the palm. Yeah. I'm guessing that's a flag that they're less mature, right? Uh, Yes, I completely agree. Yes. Yeah. Got it. Okay.

So, um, thanks for sharing kind of the stated what you're seeing, Chris. Um, so kind of just bringing it back, I was talking to Eric Monroe who will introduce momentarily. He's been on with us, I think once before, uh, as a, as a, uh, participant. And, um, I was, Eric and I were kind of talking about the CrowdStrike report, and he was sharing, Hey, you know, I got called into an incident, not a client of his just to, to be everybody to be, uh, transparent and tell, you know, Eric.

Um, but, um, this one did have ransomware involved. So I, I thought it was kind of a interesting way we could segue from Clark and looking at, you know, the, the broad based data and what's going on and how crowd CrowdStrike sees, um, kind of this, um, you know, large set data set and then go very tactical into an MSP that actually lived it with, um, a decent sized organization, uh, as, as I understand Eric.

But with that, why don't you share a little about yourself, um, and, and your MSP, uh, let's start off, by the way, with your, your, uh, world famous Americas Cup story, and, uh, we can digress from there. Well, you know, my business was kind of a lifestyle business back in the early OTs, and I was lucky enough to be on the US sailing team and the US Olympic team. So, got to travel around the world, uh, was for a brief time ranked number one.

So, uh, some of the days I'm caught working on computers. I dream of the days on the water in Miami. Sounds awesome. So, So now you way Better than running an MSP. I Love both. Yeah. So, so tell us about your MSP Eric and, and then, uh, I'll hand this over to, to Bob to get things off. Yeah. Yeah. We're, um, or Z networks. We're a, an employee first MSP in Southern California that focuses on process, policy, security and compliance.

Um, and, you know, have a really great team and we really try and make sure that, uh, we cover our other bases with our customers. Yeah. Cool. Thank you. Joining Us, we got hanging in the back there. That's a championship belt that, that he, That is, I, I have it up there for you, Gary. Thank you. Awesome. So every quarter, whichever group has the highest percentage of new MRR sales, they all get a championship belt. That's awesome. Good. Congrats. I need more of them. Yeah, yeah.

We need one on the other side. All right, Mr. Miller, kick things off if you would. Yeah, sure. So Eric, this was kind of an interesting, you know, the story we're gonna go into here is pretty interesting, right? Because I've, I've kind of been through this myself with, um, a client that wasn't our security client, but getting, getting called when the flames look at the ceiling. So this is gonna be pretty interesting.

So from what I understand, you've got a, you've got a call on a Tuesday afternoon, which turns out was about four days after, you know, the actual detonation of whatever was going on. Uh, in this case, ransomware. So, you know, like all of us, I know you go through the process, okay, well wait a minute, I'm getting called in, you know, and it's a, I don't know anything about it yet, but they're in such desperate need that they're calling in.

So you have to go in there and try to figure out what's your first, you know, you're gonna try and determine what's your first impression of that situation. Um, and then what are your, what are you gonna prioritize, you know, when you get called into to an incident like that? Yeah. My, uh, my friend called me up on four o'clock on Friday and said, Hey, we've been dealing with something. And I'm like, oh, you know, how long has this been happening? He is like, oh, since Friday. I'm like, oh no.

You know, they probably lost all of the forensic evidence. He's been trying to restore things aren't working right. Um, and the first thing I told him to do was stop everything that you guys were doing immediately please. And disconnect the WAN connections. And, uh, and then I, I asked them if they had in cyber insurance, and to my surprise, they did. So I'm like, you know, right after you get everyone to stop and unplug the wan, call your cyber insurance company immediately.

And then, uh, so then they called me a couple hours after that and said, Hey, they'd like us to come in to the, the meeting with the cyber insurance company and the breach attorney and the forensic team and the re and the, uh, recovery or remediation company too, as kind of an advisor or, or just give advice the next morning. Yeah.

I mean, what surprised me is that they hadn't actually engaged your cyber insurance organization until you say, Hey, wait a minute, you know, you need to, you need to call them. That just floors me. I mean, that, um, I mean, I hate to say it doesn't surprise me, hear it all The time. Yeah, I guess, but I mean, if you own cyber insurance and, you know, you've had a ransomware, I mean, I, you know, at least it would seem logical to me to do that. But I guess I'm like, I, Chris, you're right.

It shouldn't, Well, it, it gets lost in the insurance policies a lot of times, right? So people just, and they're, and it depends, you know, I had this conversation with brokers a lot in, in Canada, most of the brokers there have somebody that specializes in cyber. So you're always gonna talk to a cyber person, like a cyber insurance person. So they're gonna give you a better, a little bit better guidance on why you need to use that in the states.

Unless you're with a really large broker who does that. When you're dealing with more of your local brokers, you're dealing with somebody that's just general selling you all those business policies. So you're just not getting that, if we, to use our term, the onboarding that you, that we probably assume people should get to, to use cyber. So they kind of forget about it. Well, they're also minimizing, you know, what's going on and thinking that they can recover quickly. Exactly.

And understanding the extent of it. And then they don't even have an incident response plan. They haven't even thought about it. There you go. That's, that's what I was gonna say. Clearly they don't have an incident response plan. Right. There's no way you could and just ignore that completely. So, I mean, did they give you a reason why they wanted that, you know, why they waited?

I've got a couple part two part question on this one, but did they give you a reason why they waited that long or they just thought that they could handle it? They thought that they could handle it and, you know, we've got a, we've got a, uh, a part-time, uh, director of IT that, you know, can't seem to understand why his recovery efforts aren't working, but he's gonna try it one more time and it'll, it'll work best that time.

And then finally, leadership said, Hey, So what did he think recovery efforts were like? What did that look like? Uh, he actually went out and bought, you know, hard drives for every one of his computers that were encrypted. I'm just like, well, we don't really need to do that. And plus we still wanna pervert, preserve some evidence. Um, and then just trying to keep recovering from what little data they had and thinking that they can clean it out when they recover it.

And that tried it several, several times and several days and nothing was happening. Did, did e Eric, did the IT guy have any clue about forensics and the importance of it? Or is he, was he just, just Zero? The it, the, the part-time IT director is super talented technical guy that built everything from the ground up, but didn't have the 30,000 foot view of what's, of what happens during one of these events.

And, you know, kept on wondering why all of the stuff that he's done to prevent this wasn't working. So he just didn't have that, that scope of knowledge to, you know, when to kick in, you know, a real response. Was There any ego In there? Was there any ego with this guy or was he, uh, the guy was very smart and there, there is some, you know, my, I'm the it ninja type thing and my stuff's really good. Um, but it is more of Everything's great. It holds not, Yeah. Yeah.

It reminds me of that Tyson statement, right? Every plan never sur survives the first punch in the nose. Yeah. So that's kind of how this kinda works. Doesn't matter how you good you are at setting it up. It's how good are you taking the flames from looking at the ceiling.

So tell me, Eric, I mean, when you, when you got line with the, with the, you know, the, basically the, the team, the forensic team, which included the breach attorney, um, how did that kind of work and then you coming in as a, you know, as MSP to try to help support the situation? How did that all kind of meld in the early, early stages? Yeah, so it, it actually, I was surprised on how good the, the response was with the insurance company and all the other teams.

Just to go back on that IT guy, he, he was constrained by company culture and line item expense restrictions, right? So they never gave him the budget to do what he really wanted to do. So he can't, you can't put it all on his head. Yeah, that's fair. Eric just said that. 'cause the guy is bigger than him and he's, and get beat up by the guy. So he just say that to them. He must have been a really big guy. Yeah. There's not many guys that are bigger than me. I'm gonna say that.

I'd like to see this guy, Chris, If you watched the latest season of reacher on Amazon Prime that day. Nice. So anyway, So the, the insurance company was really good. They, um, the breach attorney kind of assumed quarterback, right? And introduced the forensic team and the, the, um, remediation team. And then I was surprised that they had us, you know, be the boots on the ground or the smart hands. I'm happy to do it. Good friend of mine do anything in the world for 'em.

And, uh, we, you know, we started putting together, you know, what needed to happen and who was responsible for what. And so that's, and then the, the insurance, um, the breach attorney was talking about their communications. What do they need to do with their customers, with their, with their staff, um, and you know what, what the next steps were Yeah, well, go ahead. Go ahead Eric. Maybe I missed it.

But you know, obviously now, if there's been a little bit of space, have you ever just asked your friend directly, like, why didn't you call Right your cyber insurance right away? Like, I mean, there's been comments out there, like Matt, Ty said, Hey, they don't know which is, but did he, you know, you asked the question, do you have cyber? Did he just not think of it? Or like, did he ever, did you unpack that ever?

Well, Well first they didn't, they didn't have the incident response plan to, to know what to do. Right, exactly. And then there's also a minimization of the leadership group that was in charge of it, you know, oh, this isn't too bad. We've got, we can recover with our tools. We don't need to bring anybody else in. Right. And so a bunch of denial going on and um, you know, just the lack of knowledge of what to do when something like this happens. Interesting. Okay.

I mean, it's understandable, right? Like we talk about this every day, but if you've never experienced it, you're just applying logic. Oh, yeah. And they're Expecting their DR to be, Hey, we thought we, we could recover from a flood or an earthquake, or Yeah. We had no idea. You know, that, that, you know, we would need a plan for malicious things that are gonna keep attacking us after we try and recover. Yeah.

Why don't we say that's the main difference between an it kind of mindset and a security mindset or a responsiveness? Is it, I mean, you're just programmed from the get go to get stuff back up and running. I mean, so there's no fault of that unless you've had some prior experience. So I'm, I'm with you Gary. There. I mean, just people, Hey, we need to get back up and run it.

Gar, all I was gonna say, is it, this just typifies what an, you know, for the MSPs listening, what an amazing opportunity. Like we talked about this ad nauseum, gar, but for bis, IRPs and all the other acronyms soup though about the amount of revenue at that's available, right? Yeah. As you've talked about NRR going down, there's this whole other area that is sitting at our fingertips done correctly in the sales process, which we will touch on today. Yeah. But let Bob get through his stuff.

When I get to Eric, we'll double click on kind of this view, right? And how we bring it, you know, forward. Okay. Yeah. Yeah. There's, they should take advantage of some actual free incident response training they could get online anytime they want it. There's plenty of that out there. So, so Eric, I mean, when they, but I mean the good news is, right, they could hand you their inventory, their device inventory and their routing map, right?

And then their change management log and that, that had to make it infinitely easier to manage the process, right? I mean, Zero, zero documentation, zero inventory, zero network map, 0.0, Mr. Bluda, there Was a couple of pieces of chicken scratch on a piece of paper and an estimate of how many users they had. You know, we had, we, we walked in completely blind not knowing anything about anything. Well, yeah. Well I gonna tell you, you weren't the only one blind, right?

I mean, everybody was blind at that point. Well, Bob, I'll tell you what's funny about this. I was talking to, you know, last week was Clark was on, on the call and when I, we were at, right? And boom, I was talking to about this. And what's interesting, you, you make the joke about change management, but when they're dealing with larger CL cases, the change management actually gets in the way.

They have to deploy tools and stuff, and the corporation's like, Hey, we have a change management policy. You can't just deploy tools. We have to go through our change management policy and do a change window and all this kind of stuff. So it's kinda odd hearing you say that, but we look at it from the s and b perspective, but from the enterprise perspective, it's like change management overload where you can't even move fast enough because the process slows it down.

Well, we've got a lot of huge enterprises as kind of our customer base, right? So, but I will tell you this, our, we're we're, our incident response plan is not stupid in a situation like that. We do emergency change managements and we can do those and approve those in about a minute and a half a piece, right? So it doesn't, it just means that your change management process needs to accommodate extreme situations. That's, that's all that really means, right?

But Eric, I mean, still you're sitting there, you got nothing to work with. What is gonna be your first tactical moves that you're gonna make? Uh, you know, the first thing we did was we got access to their M 365 tenant just to kind of see what's going on there. And we found, you know, basically a few hundred more accounts there than they had users. So that didn't help us all that much.

Um, we, we were able to look at their, uh, their VMware infrastructure and saw they had around almost 30 servers, you know, in the end it turned out to be they only using 11. Um, so, you know, once, so we really had nothing of real value here. And then, um, you know, the, when, when we were able to bring up an, an active directory or recover an active directory server, you know, we could see devices that were at least checked in within the last year, right.

You know, for, since 2025, we could kind of see what devices have checked in there. So they gave us a little bit of an idea of what machine names were out there, but they still had remote, remote workstations and laptops and macs that weren't registered in active directory either.

So there was a lot of, you know, just investigation and, you know, walking around and talking to department heads and, and working with whatever IT resources they had to figure out what was going on and what the inventory was. Yeah. Good. Uh, I'm, I'm gonna tell you something. I'm glad it was you and not me, Eric. 'cause that does not sound like a re any remotely at all, any kind of fun whatsoever. And this wasn't really that sophisticated attack, right?

I mean, no, from, I understand it kind of in, you know, used BitLocker and they had, so they don't, clearly they didn't have any RMMs or endpoint detection response. They, we've already talked about no incident response plan, right? A business impact analysis on which one of their vital machines. I I mean clearly all of that did. Yeah. They, they had none of that. And you know, they, they really wanted us to bring people up and running.

And that wasn't, I kept on trying to redirect them, you know, hey, we need to know what business processes are are important and then try and start bringing things up by that. Um, they had, they only had, you know, limited antivirus and no centralized management, no centralized patching. Um, and then we just, and we kept on finding, you know, this business has been there for 50 years, so we kept on finding another room or another area. Sounds like an airline.

And we had no idea what AA they still use floppy discs. Yeah, it was, it was tough. 'cause you know, every time we talked to, we find five more machines and, you know, you can't start bringing, turning things up until you have a certain percentage covered by your EDR. And so we had no idea where that percentage was. Right. And we, oh, we'll do 80%. Well, we don't know if it's 250 machines or it's 170.

So what does that percentage mean that we can start, you know, really start getting the recovery process going, This, this, this is one of those, burn it to the water line and start over. Kind of scenarios almost, right? Where you're have to do mass amputations and of networks and everything else, just kind of build up off a core, I would imagine. Yeah. Yeah. Well, it's a difficult thing. Yeah. I've been in these situations real quick, been in these situations.

It's really difficult, right, Eric? I mean, you'll probably find stuff that's old and stuff that's All over the place. And well, manufacturing, they've got systems that will, they're still running Windows seven machines that don't have updated software that, you know, they have to run this million dollar machine off this old software. So we, we kept on finding things like that and it was, it was tough. We had to, you know, there was multiple locations, right?

And then they had international locations also. Yeah. So we had to get on the phone and find, talk to their managers and, you know, get them the, the, the EDR software to install on those workstations. And, um, it was, it was, um, it was crazy. Eric was the manufacturing piece of the network segmented from the, I'm sorry. I probably shouldn't have laughed my bad. Unfortunately there wasn't. And then their, their backup network wasn't segmented either.

So all of their beam backups were encrypted had That's akin to asking do the iris drink on St. Patrick's. Let me, let me ask my Gary Peter question. It does show you there, like what you just asked. Like that's step one. Like it's a simple, it's the simplest thing. And you know, in this situation, it turns out completely differently, right, Eric? If, if there's just that bas basic segmentation, Oh man, if we had basic segmentation right, we could have restored from backups, right? Right.

Immediately. Right. And it had, and had the, we would've had backups that could have been clean, right? So the only backups that we had were, you know, storage area network snapshots that were just two days before the, they realized the event happened. So every time you kind of restored the snapshot, they got right back in. Right. And there was no clean things that we could rely on. Sorry, I just gotta ask my Gary p question. What part of this guy was smart? The IT guy?

Uh, he tried really hard and Yeah, but If you're not, Andrew, we're saying that from having all this knowledge, man, I think it's leadership standpoint. I was just being sarcastic, like, kind Of like, but listen, if you're a C-level executive at a multimillion dollar organization, I think right now the same way you need to have some understanding of, you know, finance and audit. You have to have an idea of security. Like these are all risks.

And to put all that on the person who job is to take, you know, just to put networks together. Like I think that responsibility goes to the highest level in a company. I don't disagree. But as we've seen, and we've talked about garre, um, the only time, I shouldn't say the only, the majority of SMBs Right? That are non-regulated, the when, when do they care? It's when some customer that's large that they will lose a bunch of revenue, says, Hey, here's a security audit we need you to do.

'cause you're a big supplier of ours and you need to do all these things. That's when you know, right, Eric, someone will say, Hey, we're ready to do all that stuff he told us to do. Yeah. Or when they get the, the insurance company gives 'em the questionnaire. Right. But a lot of times these big companies spend so much money with the insurance companies, the insurance companies aren't gonna give them requirements. They're like, they're, they, they're giving them a few million dollars a year.

They're gonna, they're gonna make that pathway to get that revenue to the insurance company as smooth as possible and not give those the requirements. Yeah. I'm surprised that the, i I still, I and that's true. Like there is a lot of leverage based on total insurance that that plays a part in the insurance world too. But I'm still surprised from a risk perspective, this policy was written.

I mean, knowing all that Yeah, you, you disclosed to us, and maybe that wasn't disclosed upfront, but usually that gets known pretty upfront after the claim is filed and then, you know, a carrier can kind of put it in risk mode. Yeah, I was gonna Say risk that can kind of backfire, right? If you said X, Y, and Z and then all this is being discovered. Yeah. I mean, it's crazy. Yeah.

If you're an MSP, if you're an MSP working with them and you, you help them check the boxes without actually being the case, yeah, you're in a bind. But I'm gonna tell you something I get tired of hearing, last time I looked CEOs were the chief executive officer, which means the buck stops at their desk. And I'm gonna tell you something, the flames are lick in the ceiling, I don't know, is not an answer. You're gonna wanna try and give people who are investors in that business, right?

So I I just get tired of us giving 'em a pass on not putting in the effort to understand the problem space. Yeah. It needs to happen. Put it this way, you're, you're a CEO and if, um, if something happens that like, um, somebody, your in in your accounting organization was embezzling money and they found out you didn't have checks and balances, like that's a job issue, right? Like, it's not even a question why isn't the same.

Why isn't it looked out the same of your responsibility to understand bigger risks than embezzlement, which is security risks? No, I agree. It's like the, in general, the corporate culture hasn't caught up to where we are. Yeah. It's it's so broad. There's such a big gap in education. So go ahead, Eric e Eric, real quick. They, these, I mean, I'm, I would be shocked if this is the first time they've been attacked. Did that question come up? It's the first time we know of, right? Yeah. Right.

They've been, it sounds like they're really busy trying to cover the hole before anybody knows this, right? So I mean, yeah. Yeah. So what, one of the first things we did is we put a P two license on all their M 365 license accounts, and then we put our, um, our cloud monitoring, uh, sim on it. Sure. And then we immediately found a man in the middle attack in impossible travel with that same thing going on.

I'm not sure if it was even related, but, uh, we, Yeah, it was probably just another, just another thing going on. We, we had a Porsche motor meeting with the breach attorney and they were given a list of recommendations, the standard list that we go through on this call all the time. And, and the leadership goes, well, will you send any of the list of those recommendations and the breach attorney's like, you know, we really don't like to do that.

And because if you have an issue later and it's, we have something documented that we told you to what to do, we, you know, we'd rather you just write it down during this meeting and not send it to you in official form. And the leadership said, I don't care. I want you to send me that list right now. And that the mindset of, um, not taking advice from the people that are experts, you know, is, is exactly why they've got into this, into this situation in the first place.

'cause you know, this is one of my good friends and I've been talking to him for several years about, Hey, you've got some serious deficiencies here in risk level, you really need to do something. You might also have a heroin addiction that you're not aware of. No, they're, they're focused on other things. Right. Didn't believe that this stuff would happen. Alright, Gary, over to you sir. Yeah.

That, that what you, the story you just told there, it's so interesting right into people's psyche, even after you've gotten into this situation, you're getting advice from an expert who all they do is deal in this situation and you're like, you know, just still kinda like Disregarding It. It's very dunning Kruger, you know what I mean? Uh, of not understanding kinda where you are. So, and Bob was saying they didn't have a network map, user inventory, applica, like none of it.

So how does your team, like where do you start to decide what needs to be recovered first? Like how do you put it together? Yeah. So knowing how important bis are, right? So I sat down with leadership. I'm like, well, how do you guys make money? What's the critical business systems? What do we need to do to bring up first? Well you need to bring up Jim, you need to bring up Sally, you need to bring up George. I'm like, no, no, no, no, no.

We need to figure out what you guys need to do to get operating again. And so that took a couple days, right? It really finally had to go up the food chain even longer and say, I'm not getting any information that, that we need to get your operations up in a logical manner. So they finally made everyone sit down and we went through each business process that's going on, what those systems were, what the servers were, and then who the people were that were important in each that department.

So just getting the, the three or four people that were needed to get that system running up immediately, then they can go to the next one. Um, so really tr focusing those, that leadership to, you know, to give us the information that, that what's important to them. So you're saying it took a couple days, like even to move towards some stu of like structured, uh, approach to recovery took a couple days? Yeah. Yeah.

So we kept on, you know, the, the, I was told that under no circumstances was the recovery or the forensic team need to wait on our boots on the ground, right? So then I'm like, uh, alright, so I'm bringing in, I call 'em all hands on deck. I bring in even the sales guys just 'cause they can walk around with a thumb drive and start installing stuff. Um, so we, we were all there and I'm trying to figure out what we're gonna stage in order to bring up the systems in, in the priority.

And so without knowing what those systems were and who those important people were underneath those systems, we didn't know how to stage stage it correctly. So that's when, you know, it really had to be as nice as possible, but direct and say, what do we need to get done first? And then during that whole time we've got, you know, department managers, cons, you know, high level employees pulling our engineers off to sign, well, can you get me this? Or how can I get that done?

Can I get over over here? Just distracting the team members. And then finally said, Hey, I had to pull the team together, said if anybody at all has any questions or wants you to do anything, you have to report it up to the team lead or to me and, and have them come that way. You know, the chain of command in a system, in a situation like this is absolutely important. And, um, once we establish that we are able to make more progress, Look, this is the perfect situation for a reenactment.

I mean, if you were to say, Hey, look, if I was to create a TV show like forensic files, but for these types of things, your scenario is like the perfect one for that. Chris, you know what I was thinking? Listen to Eric, what he just described the past couple minutes is literally the process that every MSP should be going through with every SMB. Look how hard it was for you to get there. Even when they're down, Eric, it was pulling teeth to get there.

Well, Eric, I was gonna ask you Gary s up and running and honky dory. Yeah. Gary, did it surprise you that they couldn't come to consensus on what business processes drove The most revenue or It doesn't not not at all. I mean, I'm, I'm on an executive team and I give, when something happens, everybody's stuff is the most important. And this is the kind of thing to take some thoughtful discussion and you gotta have the time to whiteboard some things out and gain agreement.

And that's really hard to do when like you're doing it with a gun held to your head. Um, but Eric really drives home like I'm, this is just the conversation, just a couple nuggets that you, you've laid out really almost is a training manual for the conversations that MSPs need to have with their, their customers. So in this one it halted business operations. Correct. Dead stop. Okay. For how long days? Oh, I think it was about a week and a half total.

And that was, that was unbelievably fast that we, we were predicting 24, 30 days, but we were able to, to really come. So moving forward, the way you discussed the potential financial impact of cybersecurity with your clients and prospective clients, Well the, the conversation now is more of a story, right? And letting them know what happens. And, and you know, again, most people think of a disaster as, uh, you know, how the place burns down or there's a, or there's a flood or earthquake.

They don't, they're not thinking about repeated attacks after your recovery efforts. Um, so we really talk to 'em about what it looks like and how much it actually costs. And we tell 'em that, hey, we know businesses like yours and you know, they're not thinking about these things and this is what you should be thinking about and this is how much is really gonna cost you, you know, this was well over a $1.

5 million incident for this company and who knows what it, the ongoing ramifications are with them publishing all of their data. It could have easily turned into this is a 50 million plus company dollar company. It could have easily turned into a no company, right? Because if they were lucky that they didn't encrypt all of their blueprints, their AutoCAD files, all their engineering, uh, all their intellectual property could have been wiped out.

And then what happens if that happens, if that happens? Well, before I, before I hand it over to Chris, the last comment I wanna make on this is, and this is why when you know, as an MSP and you're in front of your customers, the conversation can't be the same cus conversation of whether they should upgrade a server or buy a new phone system or whatever else like a sale. It's not a sales conversation.

This is a business conversation and you can't deal with objections the same way because obviously these business leaders don't understand the risk that they have and we're the last line of defense to make sure that they do. Like you can't come off of it, right? You can't come off of it.

And you have to make to the point where you can get where these kind of things, you almost need to make them relationship threatening for people to understand how sincere you are that this is not about them paying you more or trying to sell them something. That's only the result that they have to invest more to deal with the risks. And everyone's gotta kind of make that shift in perspective. And once you do, and the people like you, Eric, who I've seen do that, then everything changes, right?

Your customers start to listen 'cause they believe. Eric, I'm curious now that you have this story, and, and it's an interesting thing you've said, you know, Eric sent said, yeah at BIA before the story and I, and I get it, but stories are what really are important in the sales process, right? Gareth, first of all, like being able, but um, Eric, have you had time to do any postmortem yet with your friend? Like would it have mattered had we gone through this? Like what do you have?

Let me, you know, you've been talking like you said to him for years about it and now that you know, he's looking in the rear view mirror. And I liken it to, I I spent my early career doing cardiac rehab rehab in a completely different field. And I used to speak to people when I was doing rehab, walking around the, you know, little track with them after they came outta cardiac surgery.

If they told you to stop smoking the day before you were gonna have a heart attack, would you, and believe it or not, a lot of, because of it's adic it's very addictive. They didn't, but I'm just curious, did you have that, were, were you able to have yet that conversation and, and the, if so, what would he have done something different? Well, we are doing, we, we are having conversations now, um, and it's challenging to change the company's culture, right?

Um, they're all about line item expenses and reducing costs and finding the cheapest that they could get anything. Um, and so they, they're, they at least my buddy who's the, the chairman of the board or whatever, understands that they don't have the resources or capability in house, but now he's gotta get all of the people underneath him on board and change that culture. So Eric, when can I expect to look into our group chat and see that you closed this deal?

My goal is to have it done as fast as possible, but you Know, I'll be, I'm, I'm, I'm I had my alert on for it. Yeah, they, um, they're going from no budget to a significant budget now. Right? So getting through that company culture and Well, it's less than 1.5 million, right? Yes, Absolutely. Can I, Chris can I take one more because I think this is really important. This year at Write of Boom, we had Brent Adamson our keynote and he talked about his upcoming book called The FrameMaker Sale.

And that in order, the, the biggest reason deals aren't getting done is something called customer confidence. And he, he said the irony is a lot of times, right, we all think it's co confidence in our MSP and he is like, of course you have to be confident that you have a good business, but it's really about confidence that you're making a good decision and the right decision, right?

So take your friend, and it was just really interesting because he talked about one of the most important things is about, you know, the decision makers alignment of decision makers in the company. And I think that's one of the things that we often forget about that. Um, and there's a statistic in, um, his book of, of, uh, the Challenger customer and the growth of decision makers and it's gone from like one to four and now it's about 11 people plus on average in a decision.

And, and does that strike you at all since you heard his keynote and now what you heard your friend say, oh, but we have to get people on board and as Gary said, wait a minute, it just cost you $1.5 million, but he still has to get his people on board. And that, I find it fascinating, but really tell telling about sales. What are, what are your thoughts? Well, those, those other decision makers have to admit that they did something wrong, right?

And that they, that they were responsible for what happened. And so, you know, getting them past that point, you know, to being a mature leader, you know, to admit what happened and that we should have done something different. Not all leadership has those capabilities. Yeah, it, it's, it's Garrett, it's fascinating the psychology of sales and more so today than ever in, in how decisions are being made. I, I find it really fascinating. Yeah, it's, this is a good microcosm, so yeah. Alright.

We don't wanna squeeze you out. You're here, Chris. Yeah. So one, one of the questions I have is like you, you mentioned they have the international operations and sometimes what I've found is those are the ones that can surprise people even more. So even if the population impact that is smaller. Uh, you know, I mentioned this, um, something the other day, like India, they have a very short turnaround on notification compared to other countries.

And some people when they get involved in these international ones, they don't realize, well, there's some surprises. You don't have to name any countries, but were there any surprises there internationally for these guys when they had to go through this and what they had to do More of the, um, European compliance requirements? Right. Okay. So, so now they've got, now they have to deal with all that.

And I thank God I was involved in those questions with the attorneys how they were going to, to address those requirements. But yes, there was, uh, unfortunately there was, you know, PII data that was exfiltrated and you know, the, the bad actors sent them a li a directory listing of all the information that they had. And some of them now they have to go through all of those steps to address that, those issues. Yeah.

So that's really out there for the, you know, the MSP community is, I mean if, if you're a, I know some MSPs find out in these situations that their clients had operations outside the US and that's a shocker. But the other times is this, that is a way, that's another road into the conversation. Uh, because a lot of people just don't understand.

I mean, I've been in situations where, you know, Southeast Asia countries as an example, if there's like any kind of Japanese operations that's a big deal and they, they treat data, they treat the most, what we think is the most benign data as the most important data on the planet. And people don't, don't realize that.

So they Also think, so all my stuff's hosted in the cloud, but yeah, HR director's got a spreadsheet and word documents with all this every time data on it that they don't realize that they're responsible for. Yeah, every time it's, uh, yeah. One of questions I got, so when you got on the ground and obviously this environment that was full of surprises, did you find that just you, you described even bringing salespeople in to do manually.

Did you end up deploying some tools in there to kind of help with deployment and try to do just centralization? Or was it just that was more work than it? Well, one of the challenges were that, you know, none of their systems were running. So we didn't have DHCP, we didn't have DNS, so even if we put some systems in there, I thought, hey, well maybe I'll throw like a network discovery tool on there and start mapping things out.

And without all those other ancillary systems supporting it, it basically doesn't work. Yeah, yeah, that's true. And then one of the surprising things that, you know, I found out was we couldn't even kind of open up Microsoft updates, right? 'cause there's so many different URLs and ips. So it was impossible for us just to go to a machine and just restrict the, you know, we, we, we were able to open up the EDR, you know, portal so they could register into there at some point.

Um, but we couldn't do that with updates, right? So, you know, trying to make, trying to get the inventory going on with the EDR and logging every workstation into a hotspot so they could go register into the EDR and start working was very difficult. And then once we got the, um, the firewall permissions to talk to the er, none of the systems could get an IP address to even communicate with those ips. So it was, it was just a long hard sludge and crawl through all that.

How did you deal with like your team? I mean, you obviously did all hands, you got 'em all in there. How'd you deal with them in that situation? How'd they handle it? You know, kind of make sure that they got rest and people weren't too stressed out in this type of situation? And I mean, I guess I could, let's to pick on the salespeople. I mean, I don't think the salesperson woke up and that day thinking they were gonna be involved in some type of incident response situation.

Yeah, it actually, our, it turned out really good for our team, right? 'cause we don't experience this at all unless we're brought into it after the fact with a non-client. So they got really got got to get their hands dirty. You know, everyone's working together trying to get it done as fast as possible. You know, we had to do some weekend work to rebuild and re-image systems. So there was quite a bit of overtime going on. My God, that's the best sales training you can give someone. Exactly.

You Know what I mean? Uh, building sincerity and belief into the sales process. Like, I don't know that there could be a better way of adding juice and they got outta making calls. So win-win. Yeah, it really, Our team, right?

I mean, everyone was working together and knew what the goals were and they were, they saw everyone's strengths and, and what, what people did well and their commitment and showing up early on a, on a Monday morning at 5:00 AM to to load up computers into the van to bring them back to the locations and, you know, every, the teamwork was just amazing and it really strengthened our business culture. Did you have a really proud, Did you have a, Did you have a postmortem with your team?

Like Yes, absolutely. Yeah. Outside of the client? Yeah. And then I promised him a giant steak dinner tonight or this week. So looking Forward To, to that with, with everyone. We should have did the cyber call local and then we had Eric buy thanks as well. I don't think We got our invite, Gary, but always welcome Eric. Yeah, it's a late now, man. But, uh, no, that's good. I mean, um, that, that, that's you, you got a team building, you got your team building, uh, paid for as well.

You don't have to have a, you don't have to bring in a team building consultant anymore this year. You got it done in that, in that case for you. Yeah. Yeah. It was, it was a great experience for us, even though it was a lot of hard work and some stress and looking back on it, it was probably, it was, it was wonderful. I know that is, that is awesome.

Um, I think it'd be interesting if that, um, if that, that company your friend provides some type of acknowledgement to your staff too in some kind of way that would be nice as well of them.

Whether it's buy a pizza one day or do something, even though you made money off of it, you still, it would, uh, that experience, I guarantee you, is much better with you doing that boots on the ground work than if the IR firm or whomever had to bring somebody in that was completely just third party and had absolutely no exposure to the, to that company like you did. Yeah, I agree. Um, talk to us about kind of the post stuff.

I mean, you know, they had this part-time IT director and some other stuff. Like I guess there's, even after going through this attack, did they had some expectations that they just thought that those, like maybe their part-time IT director could kind of handle this going forward and didn't need such an overhaul? Or how'd that kind of go Yeah, let you know they're, they're still trying to think tactically, you know, what, what tool can I throw in here to fix this, right? Mm-hmm.

And I, I, I keep telling 'em, it's like you gotta do strategy tactical iss not gonna work. If you, if you buy a life raft and you stick it in the forward state room underneath a bunk and you don't have a to-go bag with a radio water signaling devices, it's, it's worthless, right? So if you don't have a strategy and you don't train with, with what you, what the tools or the tactical items that you have, it's, it's worthless.

So we're still talk, talking to them about, you know, what, what strategy looks like and what the funding for a strategy looks like. And, you know, hoping every day they, they, they kind of come out of their, their shell of denial of what happened From, from a sales power, from your, from your angle. How hard are you gonna keep your foot on their throats? And how often are you gonna step on their throats to get them to kind of wise up? I mean, how much, how much room are you giving them?

Not giving them on this? So with our clients, we don't give 'em any room, right? There's, we have standard, right? We, we've, we press them for bis and IRPs quite a bit, you know, 'cause that is, that is project work for us. But during our onboardings, we, we do these little mini, mini IRPs and, um, we, we have our, our tactical stuff that's required in our, in our standards. Um, and so I, there's not much wiggle room with our clients.

We just tell 'em what they have to have and we press 'em for those, those BIA, um, engagements. Um, but we do, we already have a little bit of that with our mini IRPs based upon our onboarding with clients. So I can't really put my foot on the throat of my friend 'cause they're not a client. And yeah, there's no leverage that I have over there besides, Hey, helping 'em understand how you Have the truth. What's that? You have the truth on your side. Yeah. But not everyone hears it.

Uh, Kirk, um, I'm sure you've done this, but have you asked them, are they okay to go through it again? Meaning it cost them 1.5 million? Like, and not just your friend, but this is kind of in a perfect world, it is a tabletop Hey, go ahead. Like, Are you guys real time, right? Real time, right? Yep. Well, has that question really been asked to the executive committee? Are you guys willing to go through, are you okay to go through this another $1.5 million exercise? Yeah.

So they're in process of, of interviewing other MSPs and we're gonna be included in it. 'cause they wanna make sure that there's no conflict of interest with me and a friend. So they're, they're having the conversation about what bringing in an MSP to supplement what they have, um, whether they do it or not.

You know, if they, if they budget it for it, um, if they, hopefully they can see the difference between what a company like Zephyr will do versus another MSP that doesn't, that isn't as mature as us. Um, and, and doesn't charge as much. 'cause I know they're very sensitive in the, the line item expense area. Um, well, you're saying they're gonna buy, they're gonna get an MSP but not yours. They're gonna, we don't know. They have to spend more, but they want to spend less, more.

They're gonna buy from less or more Ex the pro. Well, there's a good chance of that. I'm hoping not. Um, I'm hoping I can, um, convince them that the va there's value in what we do and why we do it. And, you know, ask Andrew's question, are you willing to go through this event and expense again by spending less? I would ask it a different way. When you go through this again, do you want it to turn out the same way? Not yet.

When, when you go through this, do you want the result to be the same or not? And, and Gary, I, I'd say, Hey, it's great that you're looking at alternatives, right? Everybody should I get it, but do you know what good looks like? Yeah. How are you gonna know the difference? Like, you know, you're about as qualified to pick an MSP, uh, as you were to deal with your security before this. Exactly. I would almost say it in that way.

Well, You know, it was interesting during this whole process, they were kept on throwing out all these tools that they were looking into, oh, we could do this, we could do that. Which one do we have? And then they kept on telling 'em, Hey, it doesn't matter what tool you get, you just have to have a process and a policy behind that stuff that you're following up on. Right? Yeah. They're not in assumed. It sounds like they're, even now they're missing the point.

They're not in an assumed breach mentality. They just think they need to patch some more holes so it won't happen again. That's the opposite of the conclusion that they should be reaching from this. That's frustrating and common. Yeah, it is. Chris, you're shaking your head. Yeah, I mean, I, it's uh, you know, I think the bigger and the older the company is that this is, this is kind of what you see unfortunately, right? I mean, it, it is just, it's it is just the facts of it.

It's just the craziest thing ever. I mean, it's gone so long and they probably have people there that have worked there forever and they're just convinced that they know everything. And I, you know, I'm not, I'm not trying to read this situation any differently, but the conflict of interest thing sounds like almost like an out clause to me. Yeah. As an excuse not to. Yeah. Yeah. I, I agree with you there. They're suspicious Too of every vendor, right?

They, they've gotta be making money that we could be saving somewhere. You already saved the money on 365 licenses it sounds like. Yeah. And Eric, I would push back with this way and say, and like, look, with all due respect, I, if I, I would say like, let me take this from a different perspective. What you, what questions, what are the right questions before you even look at MSPs? Take us outta here.

What are the right questions you guys should be asking yourselves to even make a good decision. Do you even know the right questions? Just get 'em on right now. We'll talk to him right here. Yeah, Add him on. I'm happy to talk to him for you, Eric. Get me on a call with him. Maybe we'll just show up to that. Maybe, maybe we'll have another steak dinner for them. Say Eric won't tell 'em and then we'll all be there. Be like an intervention over steak. Yeah, Absolutely. Eric.

My, my, my, one of my questions gonna be how do you choose your friends? 'cause I'm concerned about you. All right, real quick, I know we're running outta time. So we got, I think we have enough time for you to say, Hey Eric, here it goes. Looking back, what are the top three takeaways MSPs and their clients should learn from this incident to prevent them being put in the same position? Number one, one. I'm gonna say follow the cyber defense matrix please. Right? That would've saved us a lot.

Two follow industry best practices, period. There's no shortcuts. And three, understand that it is a business unit and it needs to be funded like a business unit. I know. Wow, you even got the applause. Fantastic. Alright, and then lastly, get them on a sales call with Gary. Yes. Let's get them on the tell'em. Get em on Chris. Bob. These guys could do it too. Yeah, you might actually be nicer on this one, Gary. Yeah, I might actually be nicer than you. That doesn't happen that often, Bob.

Uh, alright, everybody great. Hey Eric. Awesome. Awesome job. I think this is really important for everybody and, um, you know, wishing you all a fantastic week ahead, I'll get this summarized in a blog. I think there's some real good nuggets here. Gary, I'll get over to you. Uh, Bob Miller, Chris Laer, thanks a million for subbing in as always. Um, Gary, thanks a million. Yeah, Chris, we miss you, man. Yeah, bring me on man. I'm, I'm around. All right, you, you got it. Have a great everybody.

Thanks everybody. Thank you. Take.