When Cyber Hits the Fan: Contracts That Protect (or Expose) Your MSP

All right, everybody, it's go time. Welcome everybody, and we are back at it here midway through, uh, August. Good to see you back with us. Gary, thanks, uh, for joining and thanks all for, for out there for, for joining in with us. Um, give everybody a few minutes to get in. And while we are doing so, um, I'm gonna talk about a few things in the news.

Um, Gar, um, one thing that, you know, I um, uh, uh, was good to see was how quickly Chris, you and I were talking about this, um, last week, was enable's response to two critical vulnerabilities, um, for their end central, uh, platform. Um, if I know it's quote unquote late, even though a few days, but if you know of anybody that is running and central on-prem that may have not received updates, whatever, please, uh, let them know these are two critical vulnerabilities.

CSA did put it out on the kev. Hey Todd, great to see you. Um, and like I said, very quick response, uh, from enable getting a patch out there. So kudos. Yeah. And ma mainly I find when people are running any of the RMM platforms on prem, it's because they're, they're usually the largest customers, right? So That's a good point, Gary. That's a good point. Um, and, and funny, hopefully That means they have a team and they're on it, right? Yeah. And, and funny Gary, how things have evolved.

Like when you and I, you know, first started hearing about N Central and Nitro back in 2 0 2, 2 0 3, and how, you know, for years everything had to run on-prem, you had to have control of your PSA and your RMM. And now, now that's not the necessarily the best approach, uh, to running, to running those platforms. So that's one thing going on. Hey, Eric, good to see you. Uh, let me just close those tab.

Another thing going on, I was talking to Corey about, uh, offline is, um, a, um, this came out just this morning and you know, something Corey and I talk a lot about, are these a off base attacks? This one's a pretty significant one. Um, but the reason I wanted to put it out there for everybody to see is Workday, which is a, obviously Gary, a major SaaS platform provider for enterprises, um, was, uh, attack through Salesforce.

So their Salesforce instance, more than likely, if you read through the document, um, Corey, maybe you could kind of just give a quick overview of probably what happened happened. They just, they, they, they call out an OAuth based attack, but can you kind of walk through this probably what happened via this phishing attack? This is speculation 'cause we don't have the actual, but probably more than what happened based on what the article is sharing.

Definitely subjective with what information they provided, right? Typically, you see this, it's a phishing attack that comes in. They trick the end user into saying, Hey, we need your credentials. And that those credentials are actually creating a, you know, enterprise application for OAuth and 365 as an example.

So that way that they maintain a persistence and they get back in, they're bypassing any type of security feature you are doing because it constantly lives off that, uh, that persistence. So MFA's out out the window, conditional access out the window, doesn't matter. They maintain that connectivity. So yeah, it's common phishing tactic now. Yeah. Jamie, are you guys seeing this quite often in, in your team as well?

Yeah, I, I mean, I don't think we've seen so much from the Salesforce stuff, but we see similar things, uh, in other technologies as well. Yeah. Yeah. The, the, I mean we've, these, uh, adversary in the middle mm-hmm. Have become so common. Yeah. Um, or, you know, using some type of authentication mechanism to, to get persistence and, And it's, yeah.

I mean, if I put my adversary hat on, it's smarter because a lot of organizations, especially organizations that are trying to do their own soc right, we're, our guys are monitoring it. It's easy for them to monitor authentication requests, but most will ignore OAuth authentication, right? Mm-hmm. They'll ignore the applications that are getting created because they just don't know what they don't know. Yeah.

And so, you know, a lot of, I see a lot of organizations that just allow single sign on anywhere. They allow typical OAuth by default instead of blocking it by default. So it's a, it's a smarter tactic by the adversaries 'cause it can also go unseen for a long time. Got it. Alright.

Um, lastly, I'm just gonna put in here for today's conversation as I set the stage, uh, Jamie Levy's blog, um, that she created, gosh, about, about a week or so ago now, Jamie, I forget the exact date, but it's been out there, it feels like Months To be honest. Weeks. Yeah, it feels like months, but it was two weeks ago, actually, today ago, Two weeks today. Um, and, and you know, we're gonna talk about Jamie, what, early on, something like 30,000 views, right? I mean, it was mm-hmm.

It, it, yeah. What, what did that tell you, Jamie, that, like, when you see a blog at that kind of hit, what did, what did that maybe tell you about what's going on in the web, web universe, if you will? The internet universe? I, I think mostly people were just searching for information. They were, they, there wasn't at, at the time, there hadn't been, uh, an official release yet from SonicWall about this is what is actually happening.

Um, and so people were just trying to figure out like, what is happening. The, you know, a lot of people were getting hit by it. We saw within one week we had seen, uh, like 20 people that had basically been exploited because of this particular, uh, flaw. And so, you know, we were hit pretty heavily by it. Uh, other researchers, uh, that I was talking to in various CTI forum, et cetera, were also doing a lot of investigation.

So people were afraid they were just trying to figure out like, what, what's going on? Um, we weren't the first ones to put anything out about it, but, uh, one of the things that we thought to cover was what did pe what, what happened afterwards? Like, what did the attackers do once they gained access?

And so that was kind of what we focused on, and I think people were interested in, in that, so that they could put it in some detection opportunities or, or whatever to try to get ahead of it in case they did get hit by it, because it really, nobody knew what it was exactly, but they knew that it was serious and they wanted to be able to get ahead of it some way. And I'm sure you had plenty of time to write a blog with all the other things you have going on in there.

I, it was, that's one thing I wanted to ask you. How do you and John Hammond do those kind of things of, uh, chat? GPT writes 'em? Oh, well, Of course, Yeah. We, you can't really rely on chat GPT to, to do all that. But yeah, we luckily, you know, we have the SOC analyst who are doing the analysis. We have the reports, so we could go back and, and do that there. We did work over the weekend. Um, I, I went back and I looked at all the reports.

I did even some extra analysis just to make sure that this is correct, that we're, you know, what we're finding here. Um, and then, yeah, it took me, I, on Sunday actually, I wrote the blog and, uh, just made the graphics and all of that, but it was just putting all the, the pieces together. Um, sadly, you know, we figured out that there were actually blips leading to this before everything really started to burn down.

Um, I I, we saw pieces of this, like even as early as May, um, but we just didn't know. We, we thought these were just kind of one off sonic wall type things. But, um, but then suddenly, like after, especially right after Arctic Wolf released their report, like it really just started to go crazy. I think, I think the attackers realized that this, um, moment of opportunity might be closed to them at any point. And so they just really like set it on fire at that point.

And we, that's all we were doing for the next week or so until, until we could kind of get ahead of it and try to get customers to rotate their credentials and follow, uh, you know, all the, the things that SonicWall was putting out as like, you know, best case scenario. Um, Okay. Yeah. Awesome. And Corey, I know you've been out there since early May pre writing blogs and, and LinkedIn articles about this, so, uh, yeah. You wanna just share before I kick it off and set the stage? Of course.

Yeah. I mean, it's, unfortunately it's been business as usual. It's the unfortunate thing when it comes to this, and I'd say probably last four or five months I've had my own internal campaign, uh, where I was tired of having to respond to a lot of these compromises and finding the entry point to be a misconfigured firewall, misconfigured VPN. And it always led to that.

And so I started my own internal campaign where, you know, it's tough doing, you know, what hunters does, what we do, you have to balance the amount of alerts that you send to make sure they're actionable to make sure the partner will receive them and pay attention and do something with it. It's that fine line of alert fatigue. Um, and so I, I look at it as actionable versus non-actionable events.

And so since I was getting tired of it, you know, while it wasn't technically covered in our service plan, I was just, we were going out there just reaching out to our partner saying, we're seeing these failed brute force attempts on your VPN, we're seeing these return messages that are indicative that you have some misconfigurations, right? Hey, I'm seeing this come outta China, you have no employees in China as far as I know. Why are you allowing China to VPN in, right?

Why aren't you doing geo filtering? Or, Hey, this has already been added to our botnet blocking, why is that not turned on? And then it's just, again, initiating conversations, which is half of what we do, right, is as long as we can get them on the phone and talk about the configuration part, we can go a very long way with it. So, you know, and these definitely come in ways, but how Many, how many customers do you have?

Um, under our, so again, I'm the managed security services side of SonicWall. So those are, you know, prior solutions. Granted those that come to us for SOC monitoring or any of the MDR, whether it's cloud network endpoint, right? Um, I don't have the exact number in front of me, but it's probably total endpoints to include cloud, you know, all three pillars. That's probably, it's about half a million. Maybe. I, I'm just throwing a random number out.

So You can't really talk to them like you're saying you try to talk To, oh, as far as partners, I, I don't have the number, but it's a lot. Thousands and thousands. Yeah. Right. Yeah. And so, and, and you know, I'm not, if I see everything properly configured, if it's a, if, if it's indicative that someone's just banging on your door, 'cause that's gonna happen no matter what, right? Someone's gonna bang on your door.

If you have a firewall, someone's gonna attempt to see, okay, is there any any holes in here that I could potentially, um, that's vulnerable that I can get in through?

And so when I see all the proper responses coming out, sometimes there's not a lot you can do about it, but when I, again, start seeing misconfigurations and return messages that aren't what I'm expecting, like, Hey, this was blocked through botnet, or this was a geo filtering block before it could even attempt a username and password. Well that's, that's what I'm looking for. Got it. Alright, so let's do some quick intros. Gary, I'll turn it to you.

Um, so Spencer, you've been very quiet there and thanks for joining us. It's been a while. Um, you know, you're gonna share some perspective because from a breach side, you're working a number of these cases. So quick intro from you, Spencer, then I'll go around the horn. Yeah, thanks Andrew. Really appreciate you all having me back. Love being here. Um, as Andrew said, my name is Spencer P*****k. I'm a cybersecurity attorney with McDonald Hopkins.

Um, obviously we do a lot of the front end work, but we do a ton of the breach work groups. Done about 18,000 incidents breaches. I've done about 1100. So, um, we're pretty well versed in this, fortunately, unfortunately, I'm glad to be here to help share some perspective about that side. Thanks, Jamie. Wonderful to have you as a first time here on the cyber call. Tell us a little about yourself. Yeah, sure. Uh, I'm Jamie Levy. Uh, I'm the director of Adversary Tactics here at Huntress.

Um, so this is our threat intel arm of the co company. Uh, so, you know, we, we look a lot over these reports, put out threat reports over that, um, various threat, threat blogs, et cetera. Um, my background's mostly in incident response forensics, also software development. I'm a core developer on the Volatility project, um, which is, uh, an open source project that allows you to do forensic analysis of the RAM of any machine.

Um, I've been in this space for over 15 years, probably closer to 20 years now. Uh, but, so yeah, that's a little bit about me. Thanks Tammy. Corey, since She was, since she was 10. Yeah, Corey, good to have you back, bud. Thank you. Thank you. Corey Clark, vice President of threat operations for SonicWall, um, a SonicWall acquired our company Solutions granted November, 2023. Um, I'm actually the overseer and the architect of our security operations center globally.

So we have a, a center here in the United States. We have a center in AMEA as well. Um, I also oversee our MDR offering, so Endpoint cloud as a, as a service to MSPs and MSS SSPs. Awesome. And Chris Lara, thanks for joining. I think everybody knows you out there. I'll let you do an intro once we get to you, uh, coming up here. Gary, let me let you kick things off here on some questions. Yeah, Jamie, I, I, I'll start with you.

So, you know, we always have, um, a review of the, uh, Verizon breach report. And so in the 2025, uh, version, it said that exploitations of vulnerabilities pass phishing, right? And that's the first time that's happened for initial access at least. And so 34% rise in largely zero days associated with VPN tax. Mm-hmm. So, like, as they start to pivot from SonicWall and now back over to Fortinet, just give us some idea of where we are and kind of where you think this is headed. Oh, geez. Yeah.

I mean, this, this really isn't new. I mean, uh, the, the various apps, uh, have gone after edge devices, routers, et cetera, for, for quite some time. Uh, and it makes sense because, um, I mean, if there's an easy access to it, that's a good foothold. Uh, especially if they know that a particular device is at a company that they're wanting to target, like once they gain that foothold, then it makes it easier for them to get internally.

Uh, and so, uh, yeah, I don't think that that's something that's ever gonna go away. And I think like a, a attacking routers, especially like, uh, this is something that's just gonna continue on, especially now that people are working from home more often. Uh, this is like the lowest hanging fruit type thing. Like the, I mean, the attackers just realize this, like, uh, they don't have to phish somebody, they can just gain access probably. Nobody's even gonna know they're there.

They can just do whatever they want, listen in, et cetera. So, um, yeah, this is not an easy solution that's gonna just go away. Yeah. I mean, Corey mentioned, you know, he looks for people that have some type of a misconfiguration. I, I like what percentage of people do you think are configured everywhere on their edge properly? Oh, the that, I think so, or yeah, So low, right? I, I feel like it's probably pretty low.

I think, I don't know exactly where the onus should be, if it's, you know, like if we need to make it easier for people to configure things, like maybe that's where we need to, to head to. It's like, you know, put that, the onus on the, the, that the actual product makers to make it, you know, almost impossible to misconfigure these things.

But, um, but yeah, it just, I mean, obviously if, if that was the underlying problem in this particular case where we, you know, that we just had with SonicWall where it was just configuration issues that we, we know that a lot of people just didn't have this correct. Right. And we know that it was a big problem. It's pretty rampant. And so, yeah. So Corey, you write blogs on SSL VPNs, uh, for some time.

Can you talk us through why this is such an enduring weak point after years of patching and awareness? I mean, gotcha. There's a handful of big companies, right? And they've all been patching for a long time. So why are we still here? I mean, it's the same as Microsoft, right? It's S-S-L-V-P-N is the most commonly used technology when it comes to a firewall. If you take a step back, you have to acknowledge, okay, this is, this is entry into your front door.

And so I look as a firewall, as I've always said, at your front door to your house. They want to get in and they want to get in with as much permission as possible. And so they're targeting the VPN technology because it gives you, once you authenticate, once you're in through that door, it gives you access to everything. If you think about it like your office, right?

You're now saying, okay, you've come through the front door of my office, now you have access to the server room, you have access to everything in here. And that's what that technology gives you the ability to do. And so it gives you the keys to the kingdom once you bypass it. So, boom, we're gonna target the S-L-S-S-L-V-P-N because one, we know it's the most commonly used technology on firewalls. And two, we know the level of access it gives us is what we want and what we need.

And we've seen it, we've seen adversaries take the code and then they've dissected and the code's very, very similar across all makes, models, brands of firewalls. And it's, so I guess my question is, if that's the case, and we've been patching in for like, why are there new vulnerabilities, it seems like all the time. Do, do you know what I mean? Is it just that you're carrying old hardware and old software and just carrying legacy stuff that everyone has been evolving?

It's uh, it, it's part of it, yes. Right. So it's, it's taking, and it's, it's finding new bugs in the existing code. And a lot of it ends up being whack-a-mole. I mean, this, again, I have to again compare it to most common thing, Microsoft, we're constantly playing whack-a-mole, plugging those holes, and eventually we have to look at better technologies and what are we shifting to? Um, and that's exactly, like I said, that's exactly what we're seeing.

And to, to address the point earlier of, you know, making it easier to configure, that's tough. A lot of them, I've had these conversations with partners, a lot of 'em are check boxes, right? Oh, I didn't know that. Um, we run against, and we, we see this in our industry, right? A lot of people are like, I don't need your help. I got this. I've been in this IT field for 20 years. I don't need anybody to tell me how to configure a firewall. It becomes very difficult conversations.

Um, and this is why you see shifts, right? You see shifts in, so I'll, I'll speak for ourselves. You see shifts in us in what we're doing. SonicWall acquired A-Z-T-N-A company right after ours. 'cause long term SSL VPN's going away. I'll say, I've said it multiple webinars over the last month, right? That I, I'll bet my paycheck in three years, S-L-V-P-N is gone. If it's not, I hope no one comes to collect. But, um, this is, this is what's happening.

And then nothing Happens in three years, I've not seen anything happen. Well, You know, at that Scale in this industry in three years, I, I have high hopes. Um, and then you maybe 30 Years. I mean, a OL just announced in September. They're finally getting in way of dial up after 34 years. I didn't even know they were still around, honestly. But yeah. That's crazy. You see what's interesting, Gary, we did see Corey, and correct me, I'm wrong, Fortinet has already sunset on a number of theirs.

Oh, I think he locked up. Yeah. While he comes back a Great picture of him too. Yeah. While he comes back. And this is also a shift I'm seeing in the industry of offering specific configuration, like fully managed. Yeah. Literally got configurations, got rid of it. Um, So that's something I know that we launched and I see other, um, competing organizations launching the same thing. Hey, let us fully manage it for you.

'cause obviously we know the least, at least thing you have in your life right now is time. Right? And it's very tough in this industry to have those, those experts, uh, with that type of experience. So a lot of vendors like ourselves are saying, Hey, we have solutions where we can fully manage your firewall for you. We can send you those audit reports, those compliance reports, so that way you don't have to worry about it. It's done. Yeah.

Except a lot of time, like you said, they've been doing it a while and they know better, except that it's an issue. The same way I try to coach people are in the business and they know better, except their profits stink. It's like, yeah. Corey two FA brought was brought up, uh, in, in chat, like a lot of times we do see lack of two fa on these, you know, SSL VPNs. Mm-hmm. Talk about that.

Is it still, you know, we talked, you know, Jordan's like, oh, it was getting, we talked about earlier about getting bypassed was specifically, uh, through an OAuth attempt and that that was correct. And so you'll, you can see that, you see some vulnerabilities that can do remote code executions can completely bypass your MFA.

Um, I'll tell you from my firsthand experience, you know, I've found a lot of organizations feel they had and including claims that we saw on Reddit and investigated, they claim they have two FA enabled on everything. But in my direct experience, when I dive through it with the partner, and we're, we're trying to create that timeline, we find accounts that just aren't, you know, it sticks with me like, oh, you know, all of our accounts have MFA enabled. And I'm like, okay, well what about this?

Can you validate this specific account that was used during the compromise? And if I receive response, like, oh, that guy's been gone for three years, it's the only account that didn't have it enabled. Or we start looking at, the majority of what I'm seeing, especially in the last week, are LDAP attacks. And so what they're doing is they're compromising the service or bind account that authenticates for L dap. Now that account, it's written everywhere. And it's not just for firewalls.

This is general system admin knowledge. When you create a service account, it should have the least amount of privileges necessary. So when you're, when you have, when it's authenticating, it should not be a domain admin. But I've seen too many people using their built in default domain administrator. And that's because it's easy, right? And, and so that account has the ability to log in locally. It has the ability to do an interactive login.

And so once they get those credentials, now they're in, now they have full keys to the kingdom to do what they want. And so that was my last post that you were talking about around LDAP that I posted, and I kind of went on a soapbox rant there of we need to get back to the basics of securing these accounts. Because yes, any firewall that has this authentication account, the service account will maintain that account in the logs.

So if your firewall gets compromised, they'll be able to extract that information. So you need to reduce your attack surface. And if they compromise that account, it should not be able to do anything. So Andrew, people are staying in chat on this two F um, uh, two fa So I guess, I think what I think people are saying is, was this like what Corey said, which is they had an old account that didn't have it, or was it a bypass somehow? And I think people kind of want to drill down on that. Yeah.

Maybe Jamie, did you see, maybe, can you just see what show what you saw in all, you know, you mm-hmm. You probably quite a few of these. What have you seen? Can you kind of demystify what, what they're asking? Yeah, I think looking back in, in, at some of the, uh, once the incidents that we had, a lot of them did have LDAP issues, like what Corey's talking about here.

Um, we did have one where the customer was adamant that they had two FA, uh, so I don't the thing, the problem, like we're in a situation where we're just this third party, right? And so we don't have access to the device to verify what, you know, status. It's, it's actually in, like we, we have logs and we can kind of talk back and forth with the customer, but at some point they're gonna disengage because they had a ransomware event. And so that's taking more, more precedent.

They're not gonna, you know, come back to us and be like, yeah, we did have two FA, we've verified, here's a picture or whatever. Like, we just, we don't get that closure. And so, uh, we have to just go by what they said Happened. That's a tough Part. Yeah. Yeah. How about you Spence? Just real quick, Spence, and, you know, 20 plus cases, what are you seeing? I, you know, you get the actual forensic data, um, you're the quarterback of the case. Uh, do you hear some of the Oh, absolutely.

It was enabled and then, you know, you know it A lot of times I'm hearing, but I'm, I think His microphones, I think he has two mics. 'cause it started off Better. Sounds like He's got, he's got the shady attorney, Mike, and then he's got, Wait, better? That's better. Yes. Okay. I moved it closer. Way better. Sorry about that.

Uh, it feels like the start, I'm hearing from a lot of internal IT people that they had everything buttoned up and everything was fine, but in the past, kind of like week, we're finding out that that wasn't the case. Obviously, um, the forensics still kind of early and those are also well, internal or external, right? You know, I feel bad for the MSPs telling me that everything was buttoned up.

Um, and what I always warn people when we get on is like, don't speculate and don't make a statement that, that you're gonna have to backtrack on. Because the problem is when I've been getting on at least initially, and everybody was, a lot, lot of people were calling it a zero day.

Um, it was a little bit of an easier conversation, I think, because if it was a zero day and not for you, Corey, and I'm not saying it was a zero day, but the conversation at that time was, okay, if it's a zero day, it's almost impossible to prevent. But as we found out more, it was a lot of preventable things. So MSPs are being put in an interesting position, especially the ones that came out real for full fledged on this.

And so there's been a lot of backsliding, I think, in conversations, but we do keep seeing more and more of, I think it was a big uptick, Andrew, when we talked, uh, two weeks ago, like my group had 20, I think we've added another 10. We just got one this morning, actually. They're all Akira or they're all qua, mainly Akira and, uh, qua Locker. Um, I called a couple of the other breach coaches. They're seeing a very much the similar thing and they're seeing a kind of a similar pattern.

And I need to circle back with two or three others to find out kind of what's been going on on their forensic side. But, you know, I can give you kind of some perspective of what we're seeing with the cure when they're getting in. It seems like it's a lot less sophisticated affiliates from Akira versus the more sophisticated, which tells me that they're moving as quickly as they can at this point.

You know, a lot of times when they're getting in, they're just, they're not stealing a ton of data and their demands have been a lot lower than we're usually seeing. Um, and they're not stealing the normal amount of hundreds and hundreds of gigabytes where I think you would see a longer dwell time. I think there's a lot of more of a smashing grabs, right? Where they figured out something that they could poke around with. I have no idea how they figured it out though.

That's, and that's a lot of times that we're having trouble answering that question of how did they figure this out now? Because I know it was from about a year ago, I was talking, Chris, what Similar and what's dissimilar about the cases, Uh, everybody's blaming the m ms p. That's, that would be the biggest similarity I can give you right now in terms of the legal perspective, right? The technical side. Probably wanna write that down. Yeah, I, I'm having every conversation.

Well, the first one was like, if it was zero day, then nobody was blaming it on mssp. But it's getting very delicate at this point where some things were not enabled, um, and representations were made. I'd say that's split, right? Because I think a lot of the times, I think the MSPs were war warning clients about this and putting it in writing and those clients just weren't acting on it. And then the other 50% just sounds like it fell through the cracks.

So I don't know what's gonna go after this. Obviously. I think there's gonna be a lot of subrogation actions that could come from this. Yeah. But I think that's, for a Lot of MSPs, a lot of it they, there may be responsible for the actioning for their customers. Right, exactly. And it depends on what their statement of works is in their contract task.

So if that all, Gary, I was just gonna ask Gary, how, maybe just talk about this with Spencer on, again, this is something you just absolutely bludgeon MSPs about is process change control. How important is this right now where, you know, if you're saying, Hey, my MSA, you know, we do, we do firewall support and we do X, Y, and Z Or rather, in your scope of work rather not your MS your scope of work, how important is This piece of business?

Listen, there's a lot of areas where there's so much attack surface. I think even legally it's hard to address, you know, all of it. But this is one area where you can, right? Like it's the front door. So I think a hundred percent of the time you have to be very clear on where that responsibility in writing and in your MSA, you know, lies on this. And I, I just, again, I continue to talk to MSPs who haven't updated their MSA in five years.

Think about that, where we are compared to five years ago. So there's a hundred percent chance that this and other things are not addressed. And Spencer, I'm sure you see it all the time after the fact, right? Oh yeah. Well, I mean, a lot of 'em don't have MSAs and they have no indemnification, they have no limits of liability, which is, uh, shocking to me. Um, and uh, I'm sure Eric til if he was on this would be just cringing about that as well. Um, yeah.

And I think that opens people up to huge amounts of liability. And I'm also curious if their insurance or disclaiming them for not having those kind of contra contractual provisions in place and then opening up the insurance company for full limits of liability. And these situations are just such a stark reminder. And what I tell MSPs is, Gary, what you just said, like you need a really clear scope and really clear terms in your MSA and make sure things are very much buttoned up.

Because if not, I think people just don't wanna have the conversations. 'cause a lot of MSPs, like doctors, like lawyers, it's like long-term relationships and you don't feel like you need it. But the problem is, if one of your clients gets hit, it's probably not them gonna subrogate against you. It's gonna be the insurance company. And the insurance company doesn't care about your relationship. Yeah.

And Andrew, you mentioned about process, it starts with your MSA and then your process has to get built about that promise, right? So that's literally like, here are the things that we need to make sure that we have process in place for checks and balances automation where we can, right. In order to be able to try at least to be defensible. I don't think you can be, it's very complex, right? When you have 20, 30, 50, a hundred, 200 customers trying to keep everything the way it should be.

Even doing it right is very hard. But if you're, if you have governance, then you have a defensible, at least a defensible, you know, point. And it ties back to what your, what you promise, what your MSA says. Like, I feel like that's the best we can do. And most MSPs, you know, Spencer said they don't even have step one. Yeah, yeah, yeah. I always say, Andrew, you survive every mistake in business, right? You learn from every mistake in business that you survive. Yeah.

One thing that you hear a lot of, Gary, and maybe this is, you know, it depends on your MSP maturity, but I, when it comes to vulnerability management these days, you know, if someone doesn't have scanning and process and, you know, reviews of, of VPNs on edge, et cetera, is Oh, I've gotta go sell it. You know, first, like when you see that this is the primary, one of the primary, like I said this, the fact that this bypass phishing as the number two behind credentials.

Like what do you say to MSPs that give you that, that that's the reason I'm getting. I think you know, what I tools and think, which is, uh, you gotta decide like who knows more? You or the customer. You are the prospect. And in every case, all the security first MSPs and MSPs that we work with, this is the reason why they grow.

This is the reason why they have a higher seat price, higher profits, bigger enterprise value is because they take that stance with every prospect at every customer that they're the authority. And the only reason that the customer wouldn't pay 'em a little bit more to let 'em do it is 'cause they don't understand the risks in the same way. And that's our fault, not the customers. It's literally our job by definition is to share those risks. Now you got me all fired up, Andrew. All right.

And I'll let you go. I'll let you continue or hand to Chris Based on what No, I, no, I had one more question I, I want to get to Chris, but I think this is, uh, you know, for Jamie important 'cause it came up with Spencer several times. Like, so there were some ca call outs whether or not this was, whether it was prudent to call this a zero day, right? Mm-hmm. So hunters, uh, arctic wolf, um, you know, got ca ca called out by some people on this.

So what made you think it was a zero day initially? And let's talk about just this, this is a big deal. 'cause it's one thing if it's a zero day, it's another thing, you know, if it's not in terms of how Spencer mentioned how the customers look at the MSP. Yeah. I think that's, that's a a fair question. Um, so we definitely don't take these things lightly when we are saying that things look like a zero day. And that was really what we, we said. Like it really just, it smelt like one.

And, and also because a lot of the customers were fully patched. They, they were, uh, they had, you know, they had updated whatever. And, and so we thought this, this is probably what was happening just because of the sheer volume of incidents that we had within a week. And even prior to that, and even afterwards. I mean, it just, and then we'd also spoken with SonicWall. And at the time they hadn't figured out what the problem was.

Now they thought that it was the vulnerability that they came out with, uh, you know, later on they thought, but, but they said it didn't look the same. They said it didn't have the same rock chain and all this other stuff. So, uh, they, and that's because it wasn't necessarily that, right? It was because credentials hadn't been rotated and whatever after people had migrated away from it.

So there were, it, it was just like this weird thing where nobody really knew what the problem was, but we knew that people were getting just wrecked by it, and they had already updated to the latest version. And so that looked like a zero date to everybody. Yeah. Uh, and I mean, just how much we saw of it, Arctic Wolf, various other researchers, like I know people at other companies who are working these cases, and it, it was a nightmare.

And so it just seemed like something was terribly wrong. Like it was a zero day and nobody knew what it was. So I, I feel like it's tough, right? Because it, it carries a lot. Sometimes I feel like also people like interest, they're racing to get out there to get, to get 30,000 clicks or whatever. And it, it's tough thing. It's not the first time like this is that this is, you know, kind of pumped up and I don't really know what the answer is. Yeah. I mean, we weren't trying to get any credit.

We were just like Both MSPs and vendors get kind of caught in the middle. Yeah. Well, Can I, I get my 2 cents if that's all right. So I I call it Yeah, please. Unsubstantiated claims, right? And Hunters did say this looks like, or this might be, they never said, from what I saw, from what I read, this, this is a zero I never saw. Yeah. We never said it was, this is a zero day. Right.

But this is, this is a problem when it comes to objective facts or subjective facts and how customer bases MSPs and users, how they read it. And I have to constantly recorrect my sock on this, right? Of the separation between objective and subjective. Because if we send out an alert for any type of endpoint event and say, Hey, there's also A-S-S-L-V-P-N vulnerability included in that alert, but we're, we don't see any of their firewall logs.

Now you just told the customer, this is due to your firewall. Even though you said, Hey, there's this other vulnerability out there, I wanna make sure you're aware. They immediately jumped to, well, SonicWall socks said this happened because of their firewall, even though we don't have any of their logs. And so this is the tough part of being able to relay the information effectively so people don't just read what they want, if that makes sense. Yeah.

Um, and we did, we did investigate every claim kind of circling back here. Uh, we did investigate every claim. Uh, there were claims that said, Hey, we have a perfect configuration, um, I think include the one you might be talking about. And we got, you know, our internal threat team got on with that. And we, we found issues, we found issues with the configuration.

Um, I, I believe if I look at, we had over 200 reported cases directly to us, of those, there were two that needed a deeper dive into. Um, that's a lot. You know, that's, it's a lot of, of, you know, reports and come to find out, and it's a very tough conversation when you get on the phone with somebody and they think, and it's usually MSPs, right? As you said, uh, Spencer. And you get on, like, we're locked down, we're good, everything's good to go.

No, I mean, we looked at, when this event first started is when I started getting brought into some of these internal meetings and we look at the most common firmware version that's out there. You know, we have a, I'll just without saying the specific numbers, we have a 7 0 7 1, 7 2 and seven three. And ma the, when this first started, the majority of partners were on seven oh o. And now, as of today, most are on seven three, the most latest and UpToDate firmware.

And I mean, this kind of goes into what you were talking about earlier with vulnerabilities and patching. That's the thing that I feel like most organizations are doing the least amount of is those basics, the, the most cost effective thing you can do in security of patching. And from the MSP side, I'm seeing the exact same responses over and over. My customer won't let me. They can't afford any downtime.

Um, I even during one of these events in the last week, someone said, I can't have you isolated in my machines, my machine, my machines. I can't have any downtime. Okay? And we got this during the reconnaissance phase before an actual encryption happened. Right. We saw them running their, their typical script of what they're running before they do a full infection. And they're, we're being told, you can't, no, I'll, I'll, I'll handle it from here. No isolation. I got this. Yeah.

We can't have Downtime. We've had that on our side too, uh, where people don't wanna be isolated, even though it's like, clearly there's a ransomware event. Uh, you know, one of your machines got ransom, it's gonna spread. No, we don't want, we'll take care of it. And then the whole thing goes dark. So Then it gets to Spencer and the MSP says, we did everything right. And then it come to find out, well, isolation never happened.

It was, this was detected during the reconnaissance phase, during the initial early, early phases of this compromise. But the customer told the MSP or the Ms P assumed, well, we can't take anything down right now. So, and they, the response just doesn't happen. Yeah. This is really good.

I, I wanna make sure we get over the Chris, 'cause he's got some great questions, but this is really good for MSPs to be listening to what Corey, just like this last 10 minutes, um, of thinking about how, what you want that relationship with a customer to be based on your responsibility with that customer. 'cause it doesn't seem fair, right? Doesn't seem fair. The customer wants us responsible, but doesn't want us to do the things we need to do. Yeah.

Think, yeah, Chris, I think most people know him. But Chris Laer, solid security. Austin, incident response, cybersecurity. I got some questions, but I'm gonna throw some comments in here of the stuff I like to argue. Now, what's interesting about this zero day thing is, yeah, it wasn't a zero day, but what do we do as a community to say, Hey, you need to do something right now that you guys didn't do enough. And to get people respond.

And we kind of have these like, it, it's, today, it's kind of black and white, right? It's either it's a zero day all hands on deck or it's not at all. And I think that as a community, we need to kind of figure those things out. 'cause we do need, in this case, we did need the MSPs to jump and do something and investigate and, and take action. We didn't want them to assume that, hey, you passed your good, because that could not be the case. So that, that's one aspect of it.

Um, the other aspect is that, you know, I've been, I I I was at Evolve last week and there was a lot of conversations about this and people were coming up to me, what do I do?

A lot of people had had the conversation about S-S-L-V-P-N and I think it's one of the things that I like to reiterate to people is like, look, if you had a conver a conversation about whether or not SSL VPN is the right choice in 2025 for your client, or did you have it back in 20 thir 23 or 2022 and you haven't had that conversation again, then shame on you because there, you know, that, you know that alone.

I think there was a lot of MSPs that you could kind of tell in the conversations that we had that they had probably not addressed that particular issue strong enough. Right. And, um, and we see it, we see it on the instance that we work, we see a lot of people using S-S-O-V-P-N without MFA or with one user as it as not having MFA and everybody else does.

So, uh, you know, I always tell people, Hey, yeah, it's good to articulate risk to your client, but if they say, I accept the risk, or No, I don't wanna do that, don't just tuck that away and never approach it with them again. It's just, just not right. When I worked at the bank, our chairman and CEO, that's the number one thing he told me. He's like, look, I'm gonna tell you right now, we're gonna accept this risk, but you better not forget about it.

You better come back in six months and, and whatever, and, and, and give it to me again so I can make a decision then. So I've, I've taken that forward in everything I do. So those are kind of my, uh, comments on, on everything. It's, it's very interesting to watch this particular situation unfold. Um, I know from a damage control perspective, uh, on the SonicWall side, I just know that a lot of MSPs just said, heck, I'm done with SonicWall. And is that fair? Probably not.

Uh, but that, and I, and I, and I can say working with the SonicWall teams and the engineers behind the scenes over the last, you know, 12 months, they're incredibly responsive and incredibly transparent of what they, what they say, what they tell us, what's going on and whatnot, and all those types of things.

I mean, they've told us, Hey, look, if you think a firewall is got an issue from a security perspective, we will ship you, you ship this company a client, not a client, necessarily an incident response client. We'll ship them a brand new firewall. You ship that other one back so we can investigate and we'll tell you everything we find. So, you know, that's the difficult thing in having conversations with this to think that Yeah. 'cause another vendor is gonna be so much better.

No, no, no, that's exactly right. Not, they're not, they're not. That's my point. That's my point in this case, when a lot of, there could be a lot of unnamed vendors that would've said, Hey, this is not a zero day, I'd be, ah, but SonicWall, it just didn't, it just didn't feel right for, for somebody to say, Hey, they're, you know, they're brushing something under the rug or something like that. Just didn't, just didn't equate to me.

So anyway, with, with that, a little crystal layer se segue there, but, um, I'm gonna come to Jamie and we know that dwell time is, um, I'm gonna combine a couple of things over here. We know dwell time's very important when it comes to these things, but we also know in this latest a cure ransomware binary that they are destroying some artifacts, specifically logs.

So from your perspective, what can the MSPs do or other people do to kind of combat this or what can happen from an incident response, uh, situation to continue to do investigations if, uh, logs have been deleted? Yeah, so that's a good question. Uh, mostly you would hope that there's something that you have in place to kind of circumvent this from happening.

I mean, you know, that logs can get deleted very easily, but if you have a SIM or something where you can put the logs ahead of time, then at least you would have them to look back. And so I think that is probably where MSPs are gonna have to start looking forward is like, how could we, could we save the logs somewhere ahead of time just in case we need them? And maybe you don't save them forever, but you save them like a week's worth of logs or something like this.

Um, 'cause typically that's about what the dwell time is. I mean, even though I think Spencer said, you know, they're just kind of smashing and grabbing and they were kind of here towards the end, A lot of times they gain access ahead of time and they're just looking around and seeing what they can take or maybe taking things and then they'll come back sometime later and trash the place.

So, um, yeah, just having the logs ahead of time, like save somewhere definitely will save you a lot of heartache. Um, otherwise then you have to get really old school forensics and do like a full disc analysis type thing and try to recover them. So, yeah. Yeah. Uh, no, I agree with you. I, I would agree with they are moving faster. Mm-hmm. So we used to see 'em sitting in there, um, much longer before they started doing their xFi and, and then doing their encryption.

Um, Yeah, it used to be a, I called it low and slow, low slow attacks over the last three years were common. Now they're in and out like they were before the low and slow approach. Um, and to echo what Jamie said as well, I mean, the key thing for me around this, and again, exactly, we're seeing, you know, they're deleting the breadcrumbs, deleting, uh, logs. So what technologies does your organization have to monitor that? Mm-hmm. And then when it's happening, it's a timer, right?

As soon as you see that happening, if you can't prevent it from happening as soon as you see it happening, what is the response? And that is the importance of having an organization like us, like interests like arctic wolf, like rocket cyber, et cetera, et cetera, et cetera, of someone monitoring your environment and your solutions. Because I just hear it all too often of we're our own soc, we do it ourselves. I have engineers that do that. Mm-hmm. Yeah.

I think one of the biggest other problems that a lot of customers had was that they didn't even have full coverage. So like, even, even if they had EDR in place, uh, for some reason there's this thought of like, I'll just put EDR on the servers. That's it. I'm just gonna cover my servers. And then the attackers get in through one of the users' machines, and then they do lateral movement, and then they, you know, they're completely off the grid. We have no visibility into that.

And then the question comes back of like, why didn't you do anything? Why didn't you see anything? Because we can't protect the things we can't see is really what it is. Yeah. You know, Jamie, I gotta tell you, when, uh, over the last couple years as Kaseya launched, um, first N 365 endpoint, and then user, one of the things that I did was, you know, pre-launch, kind of make some calls to trusted, you know, customers and talk through with them.

And the one thing in both cases that I was most surprised about first with MDR in, in, um, endpoint, uh, and then in, you know, like cloud detecting response with user, almost none of the MSPs I spoke to had the sent, like I'd said to 'em, you're using Sentinel One. How come you have a thousand RMM licenses, but only 400 Sentinel one? And it's either what Corey said, they're just putting them on devices they think, or a percentage of customers.

'cause the other ones they said, don't want to, don't see the value in it. I I was taken aback, you know, by it. Yeah. Corey, quick question, Chris, uh, if I could to Corey, 'cause Corey ran on this all the time. CK West says, should we be monitoring the logs? So Corey, can you take that not only as a yes no or question, but then you know, your famous, are you gonna let me see your logs? Am I gonna have, So should we monitor the logs? And it's, so it's tough.

I know Jamie brought up a sim, but I don't see a lot of organizations feeding raw Windows logs into a SIM itself. And this is where EDR becomes so important in an environment, right? We're all used to antivirus, and by now we should all be used to the importance of EDR and then add the M on to it and someone monitoring it, management of this solution of this.

Um, so, you know, unfortunately, as we know, no matter what organization you are, it costs more money, more that you do in your onion, right? Your layers of security. It costs more and more money. So this is where you need to have, and not every organization has the same security budget. MSPs, and you mentioned maturity, right? Of course, an MSP just starting will take any business they can, right? You're doing X, Y, and Z. Yes. I'll, I'll take you on, I'll help you with that.

Oh, you're only doing z Yeah, I'll take you on. But as MSPs mature, they start driving the show here. They say, for you to do business with me, here is my solution, here is my package. And you have to get what I always call the best bang for the buck. Now, of course, I want more and more, right? I wear a tinfoil hat.

It'd be great, you know, if we can start talking zero trust on the network and the endpoint side, if we can start talking about a vulnerability management offering, if we can talk about, you know, MDR across, there's so many different layers and stacks and what you can do in your, your security. But you have to really start diving into the basics and look at the basics. Look at the numbers. Look where the majority of attacks are happening.

I always used to lead with endpoint, endpoint, endpoint, endpoint. And we know that's, that's critical, right? That's the, but I'm seeing the majority of attacks in the cloud and there's so many organizations that aren't doing any cloud monitoring. And that's a lot of times the pivot point on getting credentials, right? I mean, we talked about it when we started this about, you know, a phishing attack. I think it was the SolarWinds one. Workday work Workday, that's right. Yeah. Yeah. Workday.

I mean, and so we, it it, sorry, I, I'll go off on a rant. I'm sorry. Circling back, yes, it's important to monitor logs, but it's important to also be monitoring the right logs that get the best bang for your buck. Yeah. I have people coming to me saying, Hey, well what do you think? Do I need to put a sensor in, so you can monitor all my switches. And I'm like, well, that'll be important, but what about the stuff that kind of takes priority over that?

Are you, are you currently doing that first? Because that's gonna be the majority of things. And I'm like, no. I'm like, well, how about you save some money and someone monitoring your switches and let's look at your cloud. Let's get cloud ingestion happening. Let's look where most of the compromises are coming from. Uh, Carl Katz had a question about interest sim monitoring.

And so the reason why I brought it up is we do have a product that, uh, it, it only ingest certain logs and, and one of the most important ones that we es is the security log, which allows you to see like what, who logs on. And so we do have a lot of, uh, machine names that we track that are related to these various threats.

And yes, in this particular case, it would've helped a lot if, if some people had had some because we would've seen it ahead of time that they were getting logins from these particular malicious known actors. Um, yeah, so definitely that, that is one thing I'd recommend Chris, with time. Can we ask? Yeah, yeah. No, that's why, uh, hey Spencer, so you've been sitting on a bunch of these cases and uh, your team has been sitting on the cases.

Um, from a legal perspective, what do you see for, and I know this is a kind of a loaded question, but what do you see from a legal perspective when a threat actor gets through on the, it doesn't matter what, what the hardware platform is when they get through like an S-S-L-V-P-N and an attacker gets into the internal network.

What do you see in the, the, the biggest kind of pitfalls for the clients are, are they mainly like legal contractual or are they more, you know, class action regulatory wise? Uh, everything. But let's just start with regulatory. So what we're seeing a lot of now is the regulators are getting more sophisticated, at least in the surface level, about, okay, so what was the vulnerability?

And if we have to tell 'em the vulnerability, they're like, okay, so was there a patch available And let's just say this was from a year ago and somebody didn't patch it. The question is, well, why didn't you patch it? And if we don't have a reasonable story to be telling, we're getting more and more enforcement actions and investigations on this.

That's why like I'm highly encouraging my MSP clients to go out to everybody and one, get the statement of work, the MSA really tight, but also set expectations. So if you are as an m MSP not doing this for them, you need to make sure that they understand that, that it's on them to do it. And if you're doing it for them to do it, because regulators are all over us for this class.

Action lawsuits, definitely plaintiffs are bringing up these vulnerabilities more and more in the settlement discussions where people aren't taking the reasonable necessary steps. So I think that is impacting settlement value. Um, I think consumers and individuals as well. I think the biggest issue though, and Corey, you're right, when you were talking about this, the oversaturation of alerts, because I hear from some MSPs being like, well, we get like 50 of these.

Like what am I, you know, how many of my, these am I really supposed to be paying attention to? And my response is, well, you gotta pay attention to them. So, and 'cause what I tell clients is like, I can't go back to, Yeah, It's, but what I tell clients, I'm like, it's business versus legal and they never get along. They just business and legal always butthead. And from the legal perspective, a regulator's just not gonna accept that.

They're not gonna be like, they're gonna be like, this was pushed out, right? Corey and his team pushed this out a year ago. Why didn't you do something about it? Or as an MSP, why did you tell them that they were enabled patched? Like god forbid they do that. Um, and they're digging into the safeguards, right? The safeguards are so important here.

'cause say 10 years ago, I don't know, regulators and P of attorneys were a lot less sophisticated on this, and they were more looking on the high level. They're getting much more into the weeds now, which, you know, Chris and I have dealt with some more intensive investigations where they'll just drill into both of us about these things. Spencer, Spencer Clarify safeguards as in CIS safeguards or safeguards? Tell, can you clarify what you mean? Right, let Technical in general.

Yeah, technical in general, depending on what industry they're in. Yeah. Okay. And the other thing I was Yeah, Whatever framework. Yeah. So if you're financial services, they're gonna start pointing their fingers at that. If you're just kind of in none, they're gonna be looking for some type of general, general safeguards. And then the, the other part I was gonna add on to what my, you know, Spencer's answer is, is hey, these regulators, they're not gonna jump on week one.

You might not hear from 'em for six months. You might not hear from 'em for 12 months. So you think the dust is all settled and you're, you know, back on your horse riding in the riding in the prairie. And next thing you know that attorney general or that regulator's coming in to slap you upside the head gonna ask you more questions and more requests. So it's a serious deal.

Andrew, I, I know we're getting up against it, but I wanted to say, look, what I always tell people is this is the reason why every MSP has to determine what their minimum standard is for the tool set, for governance, for standardization, everything around IG one. Like what is your approach? And that has to be like the minimum.

And then what does it cost you to do that and make 65 or 70% gross margin so you can afford to do it and make sure that every role, every process is built into it and that's what you sell. 'cause if you don't do that, you're gonna have, if you have risk with 20% of your base, you might as well have it with a hundred percent. Do, do you know what I mean? And, and what I found is every MSP that does this, not every customer will will come that way.

They end up with more revenue with fewer customers when they go through the process and a way better company that they can build on and a safer one. And customers who want to be protected. Yeah. And we'll stay, it's better retention too. Those customers will be better, better customers. Chris Spencer, can I just, I know we might go a minute over if you guys have to jet. I get it. But I'd just like to ask you both collectively a question.

In 20 20, 20 21, we had this, basically this really reverberation from the insurance companies, the cyber insurance companies in particular around RMM. Mm-hmm. If you recall, right? There were some that wouldn't write right for a while about RMM. So this is just speculation, I just like your, both of your comments.

Could we see something similar with S-S-L-V-P-N where, you know, if you're running said S-S-L-V-P-N, if you haven't gone to some type of, you know, uh, ZTNA, we're either gonna have a higher premium or we won't write you at this level. Could, could we see a backlash with all these cases with the Avanti, Fortinet SonicWall, just ad naum in the last, you know, 12 to 24 months? What are your thoughts, Chris? From the, from the insurance or either one. Go ahead.

I would say the biggest difference right now between then and now, that was a more hard market where it was a lot, the competition was a lot stiffer, right? So 2019 we had like the five question applications, then they got hit with a ton of ransomware, 2020. So then it shifted where there's a ton of industry losses. So then they shifted into more of a hard market. Right now it's such a soft market that no, it is just, it's disappointing.

I, I won't get on a soap box about this, but I truly believe the insurance industry can be a driver of change and they can have these better requirements in place, but as long as the market, soft market is, it's easier to get insurance very right now It's easier to get Yeah. Easier to get insurance. And, and the carriers overall are competing and they wanna have the least amount of friction to try to sell a policy is the best way of putting it. So if they all, they're all gonna do it.

If one of them doesn't do it, then they're not gonna do it because they're gonna feel that that person's gonna have the edge by not asking that question or not having that restriction. So yeah, that's, is that because They figured out how much they have to charge to make money? So that's all they care about? Yeah, I mean, it, it, their business is writing policies and, and paying claims. That's their, and, but they just need to make more money than they're paying out.

And then there's a certain amount and that's what they, that's what they work on. So in the case where we had years ago with all those rms, the losses were massive, um, massive, massive times, you know, times 10 times 20 times a hundred or whatever. And so that caught everybody by surprise. I think they've learned their lessons and, and they've just figured out the risk side of it a lot better. Yeah. Got it. That's really good.

You're gonna get, uh, believe how gotta get Spence and Chris back for more. Oh, absolutely. Yeah, we can do that as early as you know, next week. Spence, sorry. No, that's great. We need get more questions your way. I, I, I thought this was, yeah, I thought it was fantastic. Thank you Jamie. Corey, you, you are awesome. Um, really, really good conversation. I thought very educational for everybody out there to our audience, thank you as always for tuning in.

Chris, thanks for, uh, you know, coming in, being a co-host. And Gary, welcome back. Good to see you. Hope we can get a few more weeks outta you here and there. Um, and wishing everybody a safe week ahead. Thank you. Take care. Thank you everyone. See you soon.