Special Guest – Chris Loehr

A beautiful Monday post Super Bowl. Um, Gary was just explaining how interested he was and exciting the game was right there. Horrible. What do you all think out there? Uh, you pleased, uh, disgusted. Um, let, let us know what your thoughts are. Um, lot going on today. I think some pe a lot, a lot of folks are in. Um, uh, I think nerdy ocon, uh, traveling there. That sounds like a lot of fun. Um, I'd like to be there. Gary's sipping some peanut clo. That's why didn't they invite us?

I don't know. But because they already are having one Gary pga, they said that was more than enough. Oh, okay. Gary Gary's there with, uh, with roost. Okay. So they said one, we can only have so much Gary pga. All right. Fair enough. Um, okay, we'll, we'll get right on into it shortly. Just a few announcements. Um, one, um, I just wanted everybody to know that, um, CSA is coming to write a boom. I think it's really cool this year they'll actually be main stage.

Um, we got the head of vulnerability management. The gentleman that you want to hear a funny story, by the way, we were on the phone, we're on a call with him, and he starts talking about sticks and taxi. Wes, the threat intel protocol, as you know, hopefully He didn't tell MSPs. They all need to start consuming it. Yeah, no, no, I Don't think he'll.

Um, but it was really, you know, so he starts going down talking about how it, it just went down this path of how evangelizing, you know, we the need for threat intel. Um, and he knew, I think you know of him too. He, he mentioned he knew you Phyllis, Tom Thomas Malar. And, um, he's like, you know, in the early days it was so painful trying to get people, you know, to adopt it.

And so we started with the financial services and me and this other guy were trying to get them to adopt it and this, and I go, you know, is there any chance, you know, Aaron Cherin? He's like, that's the guy I'm talking about. And so, what a small world. That's so funny. Um, yeah, it was, you know, he was working with Aaron and, um, rich Stru and so just a really, really small world. Um, so I'm like, yeah. He goes, I haven't seen that guy in years.

I'm like, well, you're gonna see him in a few weeks. Um, so that, and then the other announcement I just wanna make is, um, Chris, we're gonna intro you in a second. I know every little, um, most people know you, but you have a, a pre-day to a pre-day. That's right. The pre pre-day. You have the pre pre-day. Is That like being pre-diabetic? Yeah. How Does this work? Yeah, it takes a little bit of processing. Just like last night's Super Bowl last quarter and overtime period. Yeah.

You have to kind of think about it for a second. Yeah. But we do. Yeah. So Tuesday night you're around five o'clock. What do, what, what's going on? If people get in early, what are you doing? Yeah, so we're doing an incident response go figure, uh, exercise there. So I can be incredibly long, but it's going to include this time around versus just doing, like walking through a tabletop exercise, which we've done handful of times in the past.

Uh, this one's really gonna involve more of leveraging a platform alongside of that. So we're gonna be leveraging the exigence platform alongside of performing, uh, the incident response stuff to explain, number one is how a platform can be leveraged, uh, during a response, but really how that platform can be also used to prepare for an incident. And how you as a service provider can take that and provide that and charge your clients for doing the same thing with that platform. Cool.

Um, probably the biggest question and most important is if what Dustin asked is they're gonna be drinking either during, before or after Chris, That's impossible not to do in Las Vegas. So the short answer is, of course. And I might be there, right? Is that a rhetorical question? That's right. Oh, well, that's right Gary, you can just bring Sue. Yeah. Instant response. No, it'd be great to have, have her, uh, have her angle, her her perception of things on there.

But the, uh, no, it'd be cool thing. I mean, a lot of people are coming in that day anyway, so the people that have said they could come have just adjusted their flights to get there a little earlier, it is Vegas. So that's one nice thing about Vegas is most of the time you can catch earlier flights and change your flight without any type of big issue. So that's what people are doing and we're excited to see people.

Um, so it's, uh, so far the people that have, uh, said they're gonna come are different, uh, you know, different types of, uh, technical people, business people, you know, all sorts of different ranks, ranks and files. So, uh, I'd be happy to see other people join as well. There is no specific target audience we're expecting for this thing. We want everybody to come. Mike, I'll just put my email in. You can send me an email if you want. Um, and we'll get you in in there.

Alright, let's get on into it. So I made a post on Friday on LinkedIn, um, and, uh, a few hours earlier this came out that, um, lo and behold, uh, Fortinet yet again. Uh, the gear, unfortunately, you know, it's, it's frustrating. I'm frustrated for them 'cause they're a good company. They build good tech, but they have just repeatedly, um, their, uh, their, their management interfaces. And, and Go ahead, Gary. I know you No, I have a saying.

It says, you know, VPN when the first word is virtual, the next word's always a lie. That's pretty much right. Yeah. So, you know, well, yet again, um, uh, uh, nat net, you know, again, we, we've, we've tried to get people away from VPNs, obviously. Um, they've been a huge target obviously since, uh, post covid. Um, and the vulnerabilities in the Fortinets have not helped situations at all. This one now being actively exploited again.

And, and I wanted to dedicate some time to this subject, not necessarily talk about end day or, you know, they call 'em end day, or I'm sorry, Gary, we Lost you for a second. You're back. We're to stop. Gary, You're back. Just get two seconds worth. Okay. I, I just wanted to spend some time, like I said, not necessarily that this isn't about picking on Fortinet, it's really to talk about, um, this attack type of end day vulnerabilities or known vulnerabilities.

It's something we're gonna talk a lot about at write a Boom. It's something that is, uh, in the Verizon data breach report and most of the threat reports. Um, and something near and dear to Phyllis's heart that keeps getting missed over and over and over.

Um, so with that, I thought, you know, I know personally through Chris, obviously not divulging who he's working on, but over the years, Chris talking to me about, you know, and, you know, Hey, I'm dealing with this company, this MSP, whatever got popped, and it's an end day that's been out for what, Chris, four or five, six, whatever months, um, or even just, you know, a month post, which should be very straightforward to patch and mitigate. But these keep, keep coming up.

So let's, let's talk about this, Chris, and, and for those that may not know you out here, quick intro about yourself and we'll get right on into it with Phyllis kicking it off. Yeah. Real quick. Uh, Chris layer with solid security, we're based out of Austin, Texas. I'm in a second office of ours in New Braunfels, which is about 45 minutes south. Uh, we provide cybersecurity services and incident response.

Uh, most of the incident response work we do is for one of the larger cybersecurity insurance carriers. They're, they do more than cyber. But for the sake of this argument, let's just focus on cyber, uh, CFC based out in London. So we do have a London team and an Australian team. Uh, so we, you know, we deal with this from different angles all the time.

So obviously we, uh, get calls that come in with people that have been attacked, and we find out that it was reasons like a vulnerability like we're talking about today, as well as we have, you know, lots and lots of customers that we have to, uh, provide this information to and either get them or their service providers to, uh, quickly update, upgrade, mitigate, patch, whatever we wanna call it, uh, these types of vulnerabilities very quickly.

So, Chris, did you say you were in a brothel 45 minutes away from No, I knew brothels, man, that's, uh, that's disrespect to the Germans, um, in the brothels Texas. You've never been. If you come down this way, it is a great place to go. It's got a cool local music scene, local food scene. It's cool place. All right. Well, fair enough. All right, well, Phyllis, um, let's kick it on out here with, uh, with Mr. Lair. Yeah, sure. Thanks for being here.

Um, as Andrew said, um, this Fortinet VPN vulnerability was known in July of last year. Um, the cybersecurity firm, Bishop Bishop Fox, said out of nearly, um, 490,000 Fortinet S-S-L-V-P-N interfaces, um, that are internet focused or internet facing, over 69% remain unpatched. Of course, we're not super surprised yet. Still disappointed in that, in that statistic.

And then just this, just this week, the Netherlands government reported a Chinese state sponsored threat actually, um, infiltrate infiltrated, uh, its network using, um, this Fortinet FortiGate vulnerability. And so for you, you know, you work, um, doing, um, IR response. So how often were you and are you dealing with end day vulnerabilities, um, at Solis this past year? Um, that's a great question. So this past year we were dealing with them quite a bit.

I mean, the question comes up a lot is what, you know, we have people that call in kind of proactively who have an insurance policy and say, Hey, we just wanna walk through the steps in case we do get an attack. And, and one of the big questions is, is what do you see the most?

And, and I would say these vulnerabilities are what we see, uh, do the most damage so that the cases that we have that we have cause us to do the most amount of work, uh, really come back to these vulnerabilities, whether it's firewall vulnerabilities being known after they're known, after they're known. Yeah, that's right. After they're known, right? Well, you know, sometimes it's shortly after they're known, but a lot of times it's well after they're known, right?

And they've been around for a few months and we're like, wow, somebody still has this vulnerability out there and hasn't done anything about it. That's one scenario. The second scenario is, is they didn't properly mitigate it, right? So they didn't completely follow the instructions that, you know, the manufacturer had specified out there, or they didn't do it in the order.

So if we take like the Avanti secure gateway that's been out there for a little while, and they've had multiple notifications come out from the vendor and things do that, there were very, there was a very specific sequence of steps that you had to take and things that you had to do. And if you had somebody that was either not paying attention or focused, and in my opinion, when you do these things, you should have two sets of eyes.

One person doing the one, one person doing the work, one person verifying the work, uh, they, you know, they just missed a step or didn't do it. You know, with a lot of times with these gateways, not necessarily in this 40 gate, one or four to os that's what makes it kind of complicated because Fortinet has the same OS across all their devices.

But the complicated thing is, is a lot of times people will update or patch, and if that system doesn't kill those existing sessions off, and if those existing sessions are left up and running, not disconnected, or the device rebooted or whatever, you haven't completely remediated that vulnerability. So we just see a lot of mistakes done, but the most of it is, is that people put it off.

I mean, we've even had cases where you might have had a, a, a company have 15 to 20 firewalls and they got 14 of 'em, but they didn't upgrade the 15th. And you're like, well, why didn't they do that? Well, that site's really important. And, you know, they didn't want to take that side down, you know, or, or whatever the case may, you know, sense of excuses, the same excuses you've been hearing forever. That's hilarious, right? Like, think about that, what you just said.

They didn't make it secure because it was the most important. And it happens all the time. We still have ones where people have their environments get hit and they don't have any virus on their, they don't have EDR on their servers because they still go back to 20 years when some AV product blew up a server. And so that's their leadership team's policy. Don't put it on servers.

Chris, just a quick question, Phil, Phil, if I could, Phil asked, is this, you know, related to the, you know, um, vol that was last July? Phil is, what, what's unclear is they certainly relate to, they, they call out that CBE, but they call out another one with it.

So Chris, my question to you is, we don't know all the facts yet on this, but it, it seems like once things like this are known and they're, you know, Fortinet has had a multitude, it almost seems like the threat actors look to start to string things together. Is that a potential thing here? 'cause they talk about two CDEs now. One that again, yes, Phil pointing out that it was last July. But do, do you see that sometimes, Chris, the threat and Strands kind of talked about this?

Can you walk us through that a little bit? And Wes certainly chime in how threat actors might start to, you know, you know, use two of them together. Yeah, I mean, we definitely see that the threat, so a a couple of things. Number one is we really don't know if the threat actors were the first to know of this vulnerability. Whether that's something that they discovered themselves or somebody discovered and provided it to them for a price, right?

So a lot of times you don't know that, but then there's a balancing act to when okay, the manufacturer does know when that vulnerability is, and so they're going to post information about it and they need to be, there's a balancing act and Wes can, you know, add in here. There's a balancing act about, hey, this is what's going on.

So vulnerability remediate, but also not giving 'em so much information so that the threat actors can quickly deduce what's going on and take advantage of it even more quickly. You Also just put, put in, in West Western, this is, you know, a government, this is the Netherlands possible that yes, it is the one from July last July West, and they're just sitting dormant.

Because if you're in the, the a government that you know of that magnitude and that critical, you might be just, yes, that was our path in they, the log show. It was the path in, and maybe it was patched post even, but fair. I think you're mute bud. So smart of me. Um, yeah, that's exactly right.

And also, keep in mind, sometimes these vulnerabilities follow one ano, one follows another in, in terms of like, we discover a whole source of libraries may be vulnerable or, so it's not uncommon that you see something happen and then a month down the road, six months down the road, we discover there's more to the story than we didn't see. Um, that that is, it can be a common occurrence.

And then, and then also in terms of like libraries, you take something like a log four J you know, and it has, it's so resident in so many places. So sometimes we discover something, maybe resident in Fortinet, but it's actually a problem with the library that exists in in other places. And so all of a sudden it, it, it blows up. And so sometimes these things emerge and what, you know, what you think, you know, is not what you know in, in the next three to six months.

And that can become a real problem. 'cause there's a tail effect of some of these things too. Yeah. Good stuff. Yeah. Goes back. Yeah, sure. I mean, um, you know, as we know, there's no shortage of information or no shortage of data out there. We're getting alerts constantly, tons of alerts. This vulnerability, that vulnerability. I mean, anyone can sign up for any number of alerts, quite honestly. You can get them six different ways.

Um, and you know, it seems like, hey, patch, patch, patch, that's always the standard thing, right? It's pretty easy. It, it seems like a pretty obvious thing to do. And you certainly pointed out multiple reasons why, um, you know, some machines left are left unpatched. Andrew, you look like you wanna say something? I was Gonna ask you, Phyllis, with all your interaction with the, um, Ms. isac, is it a chicken and egg though? Again, we're overarchingly we are poor with inventory. Mm-Hmm. Right.

You know, is is it chicken and egg here? Yeah, like Chris said, 21 firewalls, aside from Yeah, it's an important one. Is it also that sometimes just, oh shoot, poor inventory. Yeah, I mean, I would say if you look at a lot of, um, kind of reports and, and I've talked to a lot of people, what's the number one reason why you think this company was vulnerable for whatever it was and whatever the mitigation was?

And we'll have to ask Chris, in his experience, the number one thing people say is they didn't know their environment, right? They didn't know where everything was. They didn't know where to patch it. They didn't know that data was over there. Do do and, and so know your environment is always the primary thing. I don't know, Chris, is that also valid, um, in your experience? Yeah, it's absolutely valid.

I mean, I think a lot of times, I mean you hate to put this back on who owns the firewalls, meaning like, I like the, the, the business itself, the incline. I mean, we have, we, we've seen situations where they have great MSPs, but for one reason or another, the firewalls are managed by their service provider instead, right? Mm-Hmm.

And the service provider has a very, um, monolithic way of upgrading their client's firewalls and they don't really have a process for emergency updating or, or those types of things. Or they don't have a communication process. Like in the case of this one where you could simply say, uh, you know, disable S-S-L-V-P-N, we'll, do these bigger providers have a way to call the clients and say, Hey look, we need to quickly mitigate this and turn off S-S-L-V-P-N. My bet is they don't.

And so, uh, you know, these things are, so I wanna say they, they differ from one to another, but it's just the inconsistency, I think in the, in the lack of knowledge of the in client to know what their true risk is. And, and then they don't. No one's ever talked to them about, well if, you know, these types of scenarios happen and we call you and we tell you, hey look, we need to bring the network down 'cause we gotta do an emergency upgrade.

No one's had those conversations with these people. So no one, no one's prepared. I mean, they, uh, this is a fact of life. I mean, there is some great conversation going on in the chat about just kind of Fortinet from a vendor perspective, how they don't make it easy to address these things, especially from a service provider perspective.

But, but, you know, push aside the tech technology issues and, you know, the fact that these vulnerabilities exist, there's just a lot of process issues overall with trying to handle these things. It's, it's a, it's a real nightmare once you start peeling the onion. It's interesting with Wes, what you just said, uh, there, that it's, it, it's sometimes complicated. It's almost like sometimes it's like, I've heard us talk about here, you know, security by design, right?

That sometimes just based on the way things are built. Yeah. You, you talking about my comment that was that? Yeah. Oh yeah. Right. Like, so like take web apps for example, right? Anyone here that's building a modern web app can, can update something in real time's transparent to the user, right? You, you can do rapid development nonstop, you know, something that you push into prod. It's like, oh man, this is a little weird, a little buggy. I can make a couple changes.

And a lot of times you don't even need a full, like even browser refresh, but because so much stuff is native to a modern browser, it's very easy to do rapid development. Those in those worlds. And then the operating system does not, most operating systems at that lower level do not allow for this, right? And so that goes back to the genesis of a lot of the commenting that's been happening here, right?

We're still saddled with decades old technology, especially at, um, you know, a bump in the wire solution like a, like a router or a firewall or something that's in the middle of two networks. It's just an absolute nightmare because of, um, the, the, the nature of that, that operating system. And I, I'm not enough of an expert to know how is that correctable, right?

Like, could someone develop a more modern operating system that that is, that does allow for updating and patching on the fly with no impact to actual, um, uptime. That'd be pretty awesome. And infrastructure stays around a lot longer than yes. You know, and user equipment. Well, Especially with firewalls. I mean, you look, I mean, today not, you know, we've been talking about Fortinet, but we still see Cisco.

I mean, seriously, we have cases where people have a Cisco picks firewall and you're like, wow. But you know, the thing, the things like, it's like that refrigerator in a garage, man, it'll last longer than the three refrigerator replace in the kitchen. Things just run Even though it cost $30 a month for electricity. That's exactly right. Keeps your vehicle ask.

I wanna ask you a question, Gary, 'cause Chris said, oh, you know, MSPs aren't having conversations with their clients saying if this emergency occurs, we have to reboot, right? And we need your permission to just do it on the fly versus, um, I have to call you and get your permission. So is that something that, um, MSPs, you know, from the business side, like that's a conversation. Like how is it that MSPs can do that? You know?

And, and someone wrote, um, David l asked, you know, we aren't good at doing these, this emergency patching. MSPs aren't good at that. So like, I wanna talk, ask you both Chris and um, Gary, like this is process, right? Gary Ryan Weeks always says process, it's people, process technology. What is it that MSPs can be doing, um, to make sure these processes are in face in place?

So Gary, you know, you are our business guy, um, and helping organizations set up those agreements upfront and all of that. So what do you think? Well, I'll say the first part and then the second part, I don't wanna say it, but I have to say it because I hate on every call. But it, it, it's number first off, you have to have like roles and process around this. This is not something that someone on your support desk can, can do.

It's not something that hopefully your V CIOs are spending time doing, or that's an expensive VCIO. Um, if you're gonna use your project team to do this, that's expensive. 'cause they need to meet their hours. So unless you develop some type of a, a, a, a proactive role, whether it's your knock or whether it's your techno, we call it tam, technology Alignment Manager, they're just, there's no way to get it done.

It's like saying like, we're gonna provide support desk, but we don't have support desk people. Whoever's around is just gonna pick up a call and do support desk. That sounds horrible. Not gonna turn out well. And it sounds expensive. So I think it goes to the core of, have you designed your MSP in all of these areas?

And I know that in my MSPs, um, one of the things we checked, like on a monthly basis was write down like not just what was patched, but we went right down and checked routers, firewalls, firmware constantly. Like every month we checked against it because nine times outta 10 when there's an issue with a, with a, with a router or a switch, what's the first thing that you do? Step one, you go and check the firmware and you, you check whatever can be updated and you do that first, right?

So you take 20 minutes and you do that, you know, every time. But you gotta build that in to not only your MSP, but you gotta build it in, into your offering and sell the value of that. Hey Gary, correct me if I'm wrong too. You, the, the client. Another big miss too is, you know, Brian Blakely talks about this is, you know, what's the policy with the customer? What can we, you know, what is management? You know, he always talks about management's intent for systems that are critical, right?

What is the SLA, when do we knock things offline? Because you know, it's impacting 80% of the revenue. Yep. With or without Chris, you could talk about that one. 'cause you see it in, in real life after the fact. Yeah, I do. I think there's a couple of things here. Number one is, yeah, number one is these things. Like, let's put it, I'll put it nicely when, when speaking to service providers as a whole, they like the stuff that just updates automatically or through their RMM, right?

I mean, that's just stuff that just happens, right? When you're talking about infrastructure type upgrades and throw VMware in there as, as an example. Uh, those are more involved in updating. They're not just something you can schedule in the middle of the night and things reboot and you come back the next day and see if it passed or failed. I mean, there's some work to be done. There's, there's some preparation in case something goes wrong sideways.

And a lot of, and we come back to firewalls and, and, uh, Tim Fornet, he mentioned this in here, is a lot of people don't have, uh, high availability configuration. So if you're upgrading a firewall and something goes sideways and that firewall doesn't come back up, what are you gonna do? And if you have 20, 50, a hundred of these firewalls out there for your clients and, and it's not like you wanna do 'em all at one time and they all go sideways and then you're really screwed.

So I understand the risk there and the downtime. I think, uh, that's where it's, it's, these conversations have to happen. Either you have to say, like in the case of this 40 gate one, you have to say, look, we need to update it. We're gonna do it tonight. Okay, that's fine, but in the meantime, we're going to have to cut off S-S-L-V-P-N. We just have to do it. So we we're not gonna impact everything, but we have to impact something. And so those are the types of things you have.

I've seen on the flip side, I've seen those decisions made and they end up not like, not going forward with whatever they did, right? They make the plan and for whatever reason it gets delayed and then somebody forgets about it and the next thing you know, a month later they're getting hit by it and they're like, oh man, we, we, we forgot we were put that off 12 hours and we just never got to it.

So, um, I think it's a, it's a tough thing 'cause we've been so focused on maturity of process and everything like that. And so when you take this emergency thing, people don't think that they need to have a process for that. They just think it's this ad hoc thing, but it's really not. It's a process that they have to account for and they gotta, they gotta figure out how to solve it. Awesome. Over to you Wes. Hmm. Awesome. So, uh, Chris, let's talk about tabletops for a minute.

Is there a way that we can introduce, like, you and I have done been doing tabletops since Kingdom come, right? And they've been a lot of fun. Um, you usually are the great architect and we, we bring in some awesome things that MSPs haven't thought through. I remember the last one we did actually, right?

Boom was a lot of fun, was, you know, introducing in this concept of sometimes a sales led, um, approach can, can have some real problems because, you know, the, the, the CEO just sort of let it go because it's the most expensive clients we introduced some real tricky things I think were valuable for MSPs to think about, but I don't know that we, or really, I don't have any experience with anyone introducing in, um, uh, hardware and software inventory into a tabletop.

Is there a way that you could think that MSPs could start introducing that as a way to really get clients to understand how important this is? Uh, what are your thoughts? Yeah, I I definitely think it's one to, to strongly consider and start thinking about how you can do it. I mean, the software and hardware inventory is especially important.

I mean, we've seen situations where, for example, we've seen rogue instances of remote access software and environments and credentials that the threat actor were using, threat actor was using was with a copier company that the company hadn't used in like five years. Does that make sense?

So you have a, you have a hardware component there, a copier that probably had been decommissioned, but they just took it out and didn't think about all the other things that went along supporting that copier that were maybe left behind in that environment. And then the, the same thing goes with the software itself. Uh, not having a good handle on that inventory because it was quote, legitimate software when it was installed, but it was just left behind.

And so again, somebody deprovisioned or decommissioned that copier and no one took the time to say, okay, what did, what, what went alongside this copier and the environment to help support it? So I think that's a very good and relevant example that probably people could use out there that would resonate with an MSP and, you know, their clients.

I mean, one of our first cases we ever took was a, a, a law firm, a substantially sized law firm, and it was a copier that was the source of how the, basically the malware that that got in and then encrypted the rest of the network. So that's, that's a big one. Uh, so I definitely think the hardware software, and we're seeing the software more and more now with these, uh, where you might have, let's just say a legitimate installation of Screen Connect.

And then there's another instance of screen connect in there and people just say, oh yeah, screen connect. Yep, that's a good application. That's, that's allowed in our environment. No, well, yeah, that's allowed, but is that other one, that other instance in there allowed? And that starts people going, oh, now I need to think about things differently. So there's tons of these software and Hardware. That's interesting. That is, Yeah. That, that's real.

Yeah, because I remember we, we've, I remember talking a couple years ago, we were talking about things like, if you see a foreign legitimate application, you should raise the alarm. Right? We've been saying that for years, but yeah. What about a secondary legitimate one? We, wes it's just interesting, you know, when we think about it from a threat actor perspective, they're awesome at inventory, like they enumerate, right?

Isn't that like, but we don't want to, it's kind of ironic of how a threat actor looks at the network and what they're trying to do yet. We're, you know, can you, yeah. Can you just gimme some thoughts on that? Well, They're, they, I mean, they have the advantage of, they don't have to look at it from a management lens. You know, I think almost every organization out there could do a onetime vulnerability scan and a onetime asset discovery.

And, you know, it, it's easier I think what trips the, the end user up versus the ms versus the threat actors. The threat actors just usually does it one time to say what's out there that I can go and I can leverage. It's the management. And Phyllis, you could probably speak to this better than anybody, what trips people up is the management of their inventory, right? Keeping it up to date, understanding what the process and procedure is, consolidating all of it into one big, um, uh, database.

I was talking to Joe Opat just this past week about, you know, lion guard's thoughts on all that. Not, not as a lion guard pitch or anything, but I'd love that. Like where they're doing some pivoting into some of that, because I think the management piece is where we get so bogged down on keeping it up to date and, and truly understanding what's where You want me their thought or, Yeah. I didn't know if Phyllis, you had a thought on that or not.

Yeah, that's, I was gonna type it 'cause I wasn't I if you were Oh, so yeah, I mean, I think what's interesting is a lot of times you'll see strong onboarding like that one time that onboarding, here's your device, here's your account, here's this, and then there's like week offboarding, right? It's like, oh, you always see dormant accounts, you see dormant assets, um, and it's like, like you said, the assets on the network, but how do you keep that constant tracking?

Oh, we upgraded to a different appliance, or we all our files over here on the server, so everyone, everyone wants to make sure you can get to the data even if you upgrade the equipment. But then how do you deprecate the old equipment? Yeah. And that's like, that's process. Phyllis like, yep. When you do professional services, like if you ask a a a, if you ask an engineer, Hey, how much will it cost to implement this firewall?

They'll say like, you know, x well, it's three x because part of the process is are you documenting it? Are you handing that off? Yes. Are we updating it? Like there's all these things that have to get done and they're just looking to say, oh, firewall's in we're we're done. Exactly. And it's like, you know, everyone's worried about uptime and just getting it done versus, you know, it's like, it is that, I mean it's, it's, I guess it's repetitive and it's simple to say.

It's that continuous monitoring that we always talk about to make sure, you know, you do have to, it's that keep it up to date part that everyone, you know, kind of misses in all these control frameworks and, and it, it is a lot of overhead. But, you know, as, as we see time and time again and week after week on this call, it's imperative And, and not just the process to keep it up to date, but even the logic to do something about it, right?

Like you take the, um, big colonial pipeline incident, right? It was a decommission VPN that was still alive and they just credentialed, stuffed their way in the, like, we also need to have the ability to say, well, once we have our grips around good inventory and a way to, to keep it fresh, what are we doing about it when we see Delta? You know? Right. Um, and and that's I think where we get stuck a lot. So yeah, it, It's that difference.

We talk about like what's the authorized inventory list and then the unauthorized inventory list. Like you constantly have to be checking, is this authorized or not authorized versus just there, right. It's, it's a subtle distinction, but it's important. Yeah.

I think we look too much at the, I mean, um, obviously service providers, we're, we're very focused on tools and we still have that fascination that the tool's gonna solve all these problems and the tools is gonna provide us the information and the tool may, there may be another tool to help us remediate it, but it's still gonna be people and process in the middle of that to ensure that things are taken care of and addressed and, and there's an audit trail behind it that they're done properly and successfully.

Yeah. I like, I like this statement both whenever my monthly fee was a lot more than the other person, which was always, or even on a project when we, when they said that when they got another bid on a project, I don't, we show 'em our roles and process or in a project, you know, what our scope is and says, okay, let me ask you this. What don't you want me to do? Because that's really the decision you're making, right? What don't you want me to do? You want me to not do alignment?

Do you want me to not be able to do the documentation and hand this off properly to our support desk? Do you want, like, which part? Like I'll take it out if you don't want me to do it. Yeah. But I love that it forces them to really think about, uh, what they're giving up. That's right. And shifts it away from being that price motivator. You're the same as the next guy. I love That.

And then you gotta take that to, to one more step as to explain, okay, you're not gonna do this, so here's that risk that you're accepting by not by by saying that you don't wanna do that. I think that's the, that's the big gap there, right? Is to say, look, okay, you're just making a decision, yes or no? No. It's not that simple. You're making a decision. No, great. Here's the risk. And you're by accept by saying no, you're accepting this risk. Got it. Got it. Okay, well good.

'cause I'm in a maybe a month, two months, three months. I'm gonna remind you of that risk, even though you accepted it. Now I'm gonna remind you that you accepted it and see later on if you still want to accept it. And then Chris, take it one more step and know those times when we can't allow you to accept this risk. 'cause we can't accept it. Like, this is one of those lines that we can't allow it to be this way. Like we're, we're in an impasse. That's how important it is.

And I've not maybe communicated it's risk properly to you. So we'll do it again. When you make that kind of a stand in almost every scenario, my personal situation, but also all the MSPs I work with customers will, they'll get it when they see how strong you are for the right reasons. That's right. You're strong and you just explain it and you, you come across as strong but patient, you're not gonna be able to use the same script for every one of your customers.

'cause they just are going to consume it different ways. But yes, you're right. They, they, they will get it. Or if they don't, then that's that giant red flag that you needed to say. Well, yeah, later. So last thing I'll, I'll say, and then Chris, I'm gonna ask you a different question. Roddy brought something awesome up in chat. Awesome. Um, he's, and, and Tim Fornet too. Both of 'em were talking a little bit and Roddy said that's why the change management is so important, right?

Like how do you actually know the delta of what's occurred? And again, going back to what I was talking about with Joe Opat from ING Guard, that to me an eyeopener was he said, you know, one of the things that we assumed when we started ING guard is, um, you know, we can offer good change management, but what they assumed was that MSPs had good inventory control. So they knew when Delta would happen, why it's a big deal. He is like, what we realized is that's a mistake.

Most MSPs actually don't have good change management and so, or don't have good inventory management. So if you don't have good inventory and then you see a change, you usually don't know the impact of that and how risky it is and what you should do about it. And so, again, just to channel our inner ying weeks is if you don't have good inventory, so many other things fall apart, including change management, including up keeping those up to date. Like wow. Um, so I, that's big stuff.

Um, let's, let's switch to this, Chris, what about tricky industries? So there are certain industries that just really don't have a lot of downtime, right? Like if I'm an IC industrial control system and I'm controlling the water plant for my, for my, for my, my, my township, um, if I am running a, like a power system, which include dams and like, uh, other electrical get grid producers like wind mills, like there's certain things you just can't shut down.

And so these things kind of come in, how do we deal with these in tricky industries like, especially like manufacturing, electricity, water, places like that, where it's really, really critical and it's not so easy to just schedule a maintenance window and off we go. What do you, what do you think on that? Well, I think that's where the architecture comes in.

Usually in those situations where you're finding that if you, they have like that single point of failure, let's just say a firewall to keep it the simplest example. So that's a problem, right? I mean, if it's that important and they got a firewall and it's not in an ha configuration configuration where you can upgrade one or while the other one stays online and vice versa, then that, that's a big issue right there.

Uh, or they don't have like an some alternate facility, which you can fail things over to back and forth. I mean, you know, not to resurrect that, Wes, we've talked about it a number of times here and elsewhere on a, on a business impact analysis or assessment, right? I mean, that's where it all kind of comes back to this business, this concept of business impact. And, and you have to think about that.

I mean, there's down, there's unplanned downtime and there's planned downtime and your definition of planned downtime from 10 years ago isn't gonna cut it today. Uh, it's just a different deal. And so you have to architect around that. And if you can't architect or if they don't allow you or they're not gonna spend the money to do that, then again, that's a risk that has to be articulated to them. And then they have to accept that risk. Uh, and then you have to do that.

But I mean, other people have brought in up other concepts. I mean, microsegmentation has been brought up in the chat and I've been a strong advocate of microsegmentation. The problem is, is it you're just now starting to see microsegmentation be somewhat affordable in the s and b space.

It's always been, you know, enterprise type tools out there, and then you try to trickle those things down and it's not as easy, uh, to do or it's incredibly expensive and it takes, it takes a, a pretty good knowledge worker too to, to know how to do that correctly.

Uh, but those are things, so again, that comes back to the architecture of things, in my opinion is if you're in a situation where you have something critical, like a device that is critical and from a security perspective, you need to do things and you have to schedule downtime, like there's only one weekend a quarter, you can do that. Uh, it's, it's time to rethink things and have a different discussion about 'em. Yeah.

You, you've gotta look at this with your clients from the lens of what's modern, what's happening now, and help them understand 15 years ago, 10 years ago, you had this thing and a single point of failure is fine, it's no big deal.

But now what's the financial impact to you or your, your stakeholders, you know, that can happen if you haven't game planned around, what do I do for better resiliency and if I demand and require it and there's this single point of failure if this gets owned or this gets, has to be patched, what are we doing miss client about this? We have to, we, we have to have that conversation with the client to get them to understand, um, the ramifications.

And because you're right, things have really changed in the modern era versus, um, a long time ago. And it might have been okay, well, you know, Real quick, real quick, you know, in the banking side of things, we had blackout periods for changes, right? You know, we weren't allowed to touch a darn thing in December and January because the end of year. I mean that was a, that was a big no-no. Uh, but come on. I mean, what I'm guarantee you that some of those blackout periods still exist today.

Uh, yeah. Yeah. So another tricky type situation in a lot of industries. So an in insurance question for you, since you're, you're in depth into insurance, um, just with C-F-C-F-C being, being owned, um, by a, a larger carrier. So, so talk to me a little bit more about we've, one of the things we've seen the carriers do is put their foot in the sand line in the, put whatever the line in the sand around like things like MFA, right?

Like you disclosed, you had an MFA, we discovered you didn't, we're now coming after you. There's a travelers case a couple years ago as a, a great hallmark example of that, right? Um, I'm starting to see from some of the carriers, and I'm sure you are too, that they're starting to put more lines in the sand around certain things, right?

That they, they, they sort of are saying, look, things like EDR being absolutely rolled out, things like, uh, backup and disaster recovery being mature and in place, are you starting to see them really start to put their line in the sand more deeply around some of these things?

And what about vulnerability management, do you think the day is coming that they're gonna start saying back, like, if you have 30 days something that's a critical, it's been 30 days old, why should we pay for you not patching for that? I think that day is coming. I'm just curious what you're seeing from your side of the street, Chris. Yeah, it's interesting.

And I think these kind of four, what I call, you know, four or five basic concepts, you know, MMFA backups, you know, segmented backups, all that type of immutable backups, whatever you wanna call it, having an IR plan, these things, uh, I think carriers, especially the larger carriers, had to get a handle on, okay, how can we ask those questions? How can we get responses and how can we track those at scale? Right?

It's one thing to say, I'm gonna go ask you a bunch of questions, but it's another thing to collect those responses, be able to respond to those responses. I know I, I have a client that, that their carrier just did like a, a, you know, one of those real simple external scans on them, and it was five things they found and four of 'em were not valid.

And one of them was kind of, I don't even know how they even came to that conclusion anyway, so all five of 'em were wrong, but four of them were, were completely wrong. So, and that was kind of their carrier's way of, of, of kind of checking in on 'em. But when you look at those questions, especially from someone like myself, and I'm like, wow, those don't matter really.

'cause those four questions and the things that the controls that we're looking for are important, but they're usually not the things that are related to the attacks that we see the most. So again, I think the carriers are trying to figure out, okay, how can we do these things at scale and manage them at scale and then hold people accountable to them on an ongoing basis or whatever, right? It's one thing to say, you know, a lot of times it's like, Hey, what did you put on your application?

That's gotta be accurate. How do we kind of continue to monitor that and how to what degree do we wanna do that at? Where does it become financially not feasible for us as a carrier to do that? Uh, where, where does that, uh, where does that methodology, is that methodology better than what we do with the traditional way of assessing risk on an industry and size and all those other things that we do? So that's what they're doing. But I think it's going to happen.

Um, it's just gonna be happening at different degrees, varying degrees, depending on who the carrier is, how big they are, all that, all those kind of things. Yep, Yep. Well said. Really well said. So we gotta pay attention to that. And I, I, I think that that'll be a huge boon for MSPs because clients can't solve that without, without an Ms P there, Gary. So Wes, I want to tell you that I've been on this call so long with you that I no longer say insurance.

I say insurance, It's like my southern mishmash accent coming out. Yeah, That's good. And Andrew, before I forget, I thought we agreed you were gonna hang something behind you so it didn't look like you were in a hostage situation every week. You promised me you were going to get some branding or something back there for the cyber call. Oh, you're on mute now, Andrew. Yes, Gary, I'm sorry. I I will. Okay. We'll work on That. You need help. Let me know.

So Chris, um, when it comes to patching, you know, for systems primarily, that's the function of the RMM with, you know, mobile devices. There's MDM, um, but not so straightforward when it comes to infrastructure, right? That's exactly right. It's, uh, Maybe you can talk a little bit to that. Like you can use vulnerability scanners to find some things, but then patching and updating them in an MSP environment, not so easy.

Yeah, I'm trying not to scare everybody, but I mean, there's been some great conversation earlier in this thing about, you know, what it takes from the Fortinet side to do that. And there are ways of doing it, but it's very expensive for MSPs to do that, right? If you're a in client with a bunch of firewalls, you handle it internally, the licensing for that is much more favorable for you than it is from a, from a, from an MSP's perspective.

If you're using for to cloud or Ford manager or a combination thereof, not gonna get into that. But, uh, for all those platforms, it is, it is difficult, right? I mean, especially when you're doing, whether it's server hardware, network hardware or whatever, it's a, it's a, it's a nightmare. Whether it's phone upgrades, if you have phones sitting on tables or desks and, and doing those things or you know, all the video conferencing equipment and everything.

I mean, there's just tons and tons and tons of stuff to have to do that with. And again, a lot of it is not as friendly and as automated as you have been spoiled with an RMM. The other thing is we're kind of the, It's not always the same though too. Yeah, it's, I mean, I know we would get a client, like the firewall we would always rip and replace, like that was part of our process. But not every switch, not every router, depending on the client was not feasible.

Um, so now you have that complication on top of it. Yeah, you do. And, and I, it's, it's, uh, it's one of those things where it's, I can understand the challenges where you're like, Hey, it's not with patches and, and those updates, it's a recurring thing, right? So it's very easy to understand, hey, this is part of our service because this is something we're gonna be doing all the time.

'cause there's all these security updates and patches coming out for operating systems and applications, but with network equipment, it's not, it's not so, right. I mean, I always used to get, use the examples of printers. I mean, print updating printers and those network interfaces on printers is always a nightmare. Doesn't matter what tool you used, even internally. And then you talk about an MSB, having to do it for multiple clients, it's a nightmare, but they just don't do it.

And they put it off, they put it off, they put it off, and then one day, you know, those things are always a low vulnerability or a low risk, and all of a sudden one day they become a high risk. And if you would've taken care of it two years ago, you wouldn't have been in that situation. So it, it's a challenge. We've gotta figure that thing out and then have, again, charge accordingly to, to make sure that that's included in your offering and that you can handle it.

And if you, that could also be the impetus behind you explaining, Hey, you do have some gear here that is working. There's nothing technically wrong with it from an operational perspective, but from us, our ability to be able to manage it effectively, especially from a security perspective, it's gonna be much more expensive than us just replacing it with something more modern. Yeah. So I think that's, um, that's a interesting point too.

But I also think that the vulnerability, vulnerability scanning for as good as it is, it may not necess, you know, it's not perfect. And a lot of times people assume, I mean, a vulnerability of a vulnerability scanner can tell you that whether something's updated or not, but it doesn't necessarily tell you that it was, it was thoroughly or, or, or done correctly. Right? And that's our Example day that you used is A good one. Yeah. Right.

See that's where you have to have, you know, certain levels of testing we haven't really talked about that I'm aware of lot is inside out testing, right? I mean, I remember years ago, Andrew, when we went, we, when we went up there, um, to NIST and we went into that IOT lab and they were talking about, Hey look, this device is fine, this device is fine, but when these two devices from two different manufacturers are on the network together, it's bad. Mm-Hmm.

And so those are the things and those combinations of things, those are the things that we still need to solve for, especially at the MSP and, and the small business side. 'cause those are, those are real threats, real issues, real risks. Yeah. I was gonna say, Andrew, one, one point I wanna make on this is I was having a conversation with a pretty big MSP and I was asking them about their proactive approach, right?

And I was talking about how what we teach around technology alignment manager, and he said, well, you know, we're at scale, so we do it a little differently and it's part of our knock. And we have a team that does proactive services.

And the point that here is, unless there is someone who's responsible for some number of accounts, 2030, where they have accountability on all this little tiny one-offs in detail, I, I, I, I don't know how a team that's sliced up into verticals goes to, you know, you know, hundreds and hundreds of customers and thousands of details and, and doesn't miss the kind of things.

Like at some point someone has to be accountable for some number OO of your customers on this to know what good looks like and to be able to audit it at that level. And if not, there's just, every week we bring up more of this, Chris, things like, this is hard. This is like, some of 'em are simple, like, you're not doing something like this. But so many of 'em are like what we're talking about today. Like we're identifying it and we're saying this is not easy. Right.

Like someone's gotta be accountable. Yeah. Right. Garrett, question to you, MSP's getting to scale, whatever, you know, term. Let's just air quote scale. You know, I know for you, with some of the boards you sit on, you look at splitting up RMM based on supply chain risk. Bringing this back to this, you know, you're an MSP, you've got hundreds and hundreds and hundreds of client on one side. You look at standardization and the importance of it.

How do you, how do you balance supply chain risk with standardization? Is it, is it a real thing we're getting to with these larger MSPs? Yeah, and uh, from my standpoint, the way I look at it is that, uh, supply chain risk trumps efficiency. Okay. Like you have to, I, I saw someone talk about blast radius, right? You have to decide what your pop potential blast radius.

I mean, I, I work with some companies that I feel so strongly, uh, that an event that affected every customer is the only thing that stands between where they are and getting to the end result. So it's that, to me, that's the most expensive thing. But if you think about it, Andrew, you know, MSPs, even if you're smaller, if you're blast radius as your whole Ms p, that's big.

That's just as bad on like, on a, like the impact to that MSP owner is the same as some, probably worse than some big private equity company. Probably is better to get through it. So, so for an m even a smaller MSP Gary, and it's, you know, the 80 20 rule, let's just say there's 10 big customers. Do they need to maybe look at the 10 largest MA making up 80% of their revenue? How they look at the risk of those 10? Yeah, many do and many do that already. Okay.

But it's always gonna be a balance right? Between efficiency and you gotta do it in a way. 'cause if you add too much complication, that can be a security risk. Yeah. Okay. Whole thing's a balance. Chris, um, uh, Mandy publishes, uh, uh, uh, stats around time to exploit trends. And with ai, um, bad guys are reducing that. They're getting to things more efficiently and faster. Like, is that an advantage, like the new technology?

Are they able to use like things like AI to, to, to, you know, to get to exploits faster compared to MSPs using it to stop things faster? Or is it a disadvantage? Yeah, I think it's a, it's a race. I mean, the one thing about a lot of these threat actor groups is they're very well organized and they have very smart people. Now, a lot of them are trained, educated, security people. These aren't just people that woke up one day going, I wanna make a bunch of money.

I'm gonna go into hacking, uh, these people are at. So I can definitely guarantee you that they're leveraging every tool, including ai, uh, to do what they need to do. Whether it's to take advantage of an exploit, help automate the attacks on that more quickly. Uh, just like MSPs are trying to use AI to automate stuff on their side, that threat actors are doing the same thing. And I think they're also doing a, a much better.

And I think especially now, these groups, these organizations are doing a much better job managing their people, if that makes sense. Uh, managing the associates and, and doing those types of things as well. And I wouldn't be surprised that they're using AI and some for, for the business knowledge that AI can provide you to do that as well.

So I think, uh, uh, no doubt that they're using the same AI tools as that everybody else has access to, along with probably some other stuff that's being developed, uh, specifically for their purposes. So, I mean, you have to do it on both sides. Yep. Um, you know, you get run over. So, and we have like five minutes left, so it's not often we get you here, which we'd like to get you more often. Um, but you do incident response.

So can you give us like a little update on like what your state of the industry is from your unique lens that you see things through? Yeah, and I, I know I haven't been on, and I think, uh, it is good timing because I will tell you that, uh, I would say in the last 60 to 90 days, we are starting to see it MSPs attacked again like we did a couple years ago.

Um, I'm not saying that they're necessarily targeting, but I do see where they're hit and, um, there's no evidence that they were necessarily targeted, but once they got in there, the threat actors figured out, Hey, there's more to this than just a simple single company and we're gonna go after the others. Uh, so definitely seeing that we've seen, you know, attacks, there's been, there's been ones in the news.

I, it doesn't matter whether you're just a standard MSP or an MSP that's also hosting systems. Uh, but we've seen some MSPs that are hosting systems for all their clients and they get hit and basically all those systems are down as a result and it's really, really ugly. It goes back to, you know, microsegmentation how those things are architected, you know, what type of credentials. Uh, we've seen cases where MSP's credentials have been compromised and, and used against them and their clients.

Uh, so I, it, I don't ne I can't tell you a root cause of this, uh, I can't tell you that. Have we, are we starting to see MSPs be more complacent again and, and not as, uh, being sensitive as they could have been when there was a lot more people in the room that were sharing examples, uh, that they've been attacked. Uh, but we're starting to see it more and more.

And I would think that, uh, uh, also with some of these groups just take, uh, not to give 'em any credit, but black Cat, alpha v uh, those guys have said, Hey, the gloves are off now. We don't care what industry and who you are, we're just gonna go, go get you. And, uh, so that, that's the scary part about it. So definitely, uh, I think we're gonna see more MSPs attacks. That's my kind of my prediction in 2024.

Uh, def definitely, I think we're gonna see some more of these types of vulnerabilities that we've been talking to today, and we're gonna see those things, uh, be taken advantage of more quickly. And I just think that, um, I think the, you know, the, the, the payments, you know, and the other thing we've seen is that ransom payments are still occurring at a pretty high rate. I think I saw a number the other day. It was like 1.3 billion or something like that last year, uh, was paid out.

So we're starting to see these threat actors take more and more data when they get into a network. Uh, you know, it used to be, you know, some gigs and a couple hundred gigs. Now we're seeing terabytes and we're seeing, and we're not seeing a portion of file servers. We're seeing the entire contents of file servers taken. So that's forcing people into situations where they never thought they would have to pay, but they're gonna have to pay to protect that information from getting out there.

So, yeah. Um, the, you know, the sad news is I don't see it getting any better. You're just a a, a bright light Wish it was getting better. I really do. Yeah, I know you, I I know you do. So, uh, Andrew, I'll kick it back to you. We'll get some final Yeah, I'll just recap it with this, Chris. Awesome. Have you, I mean, I don't think it's surprising, right?

When you look at the threat acqua profiles, they, they, the major ones in their, who they're attacking by industry tech, they technology, you know, service providers is listed in almost all of 'em. Again, I'm air quoting, right? But if you bear, if you look at the, you know, MITRE attack and you look at the, the top ones they're, that's in there. Um, Wes any closing comments on that?

I mean, that's what I remember seeing in the, you know, Ryan's doing a metadata study right now, um, for, for obviously for right of boom, but technology providers are because of the supply chain. I, I gotta believe. Fair. Yeah. I mean we're, we're, we've seen little snippets of it here and there. Um, and, and maybe it's the plethora of vendors.

Maybe it's the lack of insight into how some of the modern platforms may, maybe they're just, uh, there's a lot to say that could sound like conspiracy, right? But, but I I, I, we haven't seen it yet. I hope we don't. Um, and, and, but, but you do wonder, right? You, you certainly do wonder. And it, it's certainly something we should all be concerned about, especially for those of us in the vendor ecosystem understanding how many, um, MSPs we serve.

Um, we should all take it far more seriously than every MSP we, we should eat our own tacos. Yeah. Very good stuff. Alright. We're right. Awesome. Today. What's that again? I said Chris, this was awesome today. Yeah, it was. Thanks. Appreciate it. Appreciate having me on. See you soon man. Man. This was some great discussion on the chat day though. I mean always, always, but this one's pretty, yeah, you almost have a look at that one. Yeah. Alright, everybody, have a fantastic week.

We'll look forward to seeing you next Monday. Thanks, everybody. Talk, talk soon. See everyone. Thanks.