03/15/2021

All right. Welcome everybody to week 41. Joined as always with Wes Spencer, Ryan Weeks and Gary Pika. How's everybody? Hey. A little tired, A Little tired. Wes, were you dealing with exchange exploits over the weekend or, well, Less me and more my team. Uh, Okay. But Yeah, we've been busy with it, as I'm sure others on this call have been as well. I'm sure. I'm sure. Well, hey, we have one of our favorites with us, Chris La Solace, security, now CFC Underwriting. Chris, welcome.

Hey, thanks from Thanks. I prefer, I'm happy to be here even in, uh, in, despite the circumstances. Yeah. Gary, I don't know if I should worry about this or not, but my girlfriend said Chris is on today. I'm not missing this one. Yes, you should. WI not get this comments every quarter. Right. You know, it make you wonder if I come to Tampa, what's she gonna say then? But she never says that about the three of us guys. Yeah. Well, you guys have the glasses. Chris has different ones.

He's got the lighter. I started the movement. Let Me throw up a quick poll here, um, and start things off. We'll make that one visible. We've got a few polls. Um, here's what, you know, just a little bit of the setting the stage. Um, I have the very good fortune of having a friend like Chris and to get to talk to him a lot, uh, each week. And it kind of made me think about what just went on and, you know, we've all been talking about it collectively, the hosts.

And just like Gary, you know, you and I were chatting offline like, man, what if it was 20 15, 20 14? And we weren't quite that far migrating people to Office 365 Or, you know, what if, or you said when, when, when Office 365 has a mass exploit. It made me think about, 'cause Chris is like, Hey Andrew. Uh, we had 80 cases last week. I couldn't even respond to people. Like, not that they didn't respond, but like, I'm like, Chris, what if there's an msp? He's like, well, they're in the queue.

They called me. I'm like, what on caseload, Chris, five, seven? You know, something like that. And I'm like, wow. So if you have every IR firm book, which they were, right, Chris? Pretty much. Yeah. Yeah. I mean, it was, I had never experienced anything like this situation, ever. Right. And I mean, this goes way back, right? I mean, we've had, I mean, we've had the SQL stuff in the past, we've had all sorts of stuff. This is just, um, a completely different situation altogether. Yeah.

And Chris, Do you, do you feel like this is a culmination of things? Or do you feel like this is literally just a, um, happenstance, a circumstance? No, I feel that it's a culmination of things. And what I really feel is that the, especially the, um, if you wanna call it the hacker industry or the hacker environment is ridiculously competitive right now. And they're all, it used to be just pretty easy things to go after.

Right now, I think it's just, there's so much going, there's so many people going after it. So there's money, obviously large sums of money involved, but there's also this reputation stuff. And I just think, and then ob obviously the nation state stuff that just kind of is just an under, you know, a little sub level underneath all that. I just think there's just tons of parties going after it.

I mean, it's just, I guess it's blood in the blood in the ocean, and the sharks are in a feeding frenzy, and it's just gonna continue to just, these things are gonna happen and, and there's just too much money and too much stuff involved for it not to, I mean, think about it. That's a super interesting perspective, that this is less about SMBs and MSPs, that's collateral damage. This is about a new industry, uh, a burgeoning industry that's maturing, that's competitive, just like MSPs.

Industry's changing and competitive. And just, just so happens that at this point, MSPs and SMBs are just in, in the way, it's nothing personal. And in this case, yeah, I agree. I think it's nothing personal. I think there are some other groups that obviously target MSPs because they know the, you know, they know how much value there is to doing that. But this one was, I think you have this group half Nim that had their own particular motivation to do this.

And when they got it just opened up the doors immediately. And obviously we don't know how that information was leaked, and there's all these theories and all this kind of stuff. But in the end, I mean, once this thing was revealed, I'm sure everyone in that community was just angling to get their hands on this thing. Because I mean, it was the, so, So let me just, uh, go back just for a second, Chris. Not everybody, um, knows you, I think a lot of people do out there.

So let me just kind of start off by, could you give a little bit of an overview of yourself, solace now, CFC and, and what you might do, uh, each and every day? Um, Yeah, I was just thinking your girlfriend could do it better than I can. Oh, man. Anyway, no ware solid security.

Also, I mean, we really know that solid security and CFC response, the only time you'll hear us ever talk about CFC is if you're, you know, there's a, there's a claim involved, but, but traditionally everybody knows it's solid security. We've been around 18 years. We were acquired about a year and a half ago. Uh, started just doing cybersecurity for banks eons ago, mainly community banks, and then grew after that.

And a few years ago we got into the incident response side of, of things, and that took off like a, I guess, um, Elon Musk rocket and everything went off from there. And so, uh, you know, today I spent a lot of time on, you know, on both sides of our incident response than our cybersecurity side, but a lot more time really on the incident response side, obviously, if you wanna call it fighting fires or whatever.

And so, you know, we see a lot of stuff, and most of the time I've been on these calls and I enjoy every time I do it, you know, we're talking about how to help MSPs. And most of the time it comes up with ransomware cases, right? And there are some ransomware obviously, um, relationships here with this particular event. But, but this is, you know, kind of a completely different thing, really. Um, you know, I guess something that us on the security side have been expecting.

And what's interesting is a lot of people in the security space are like, Hey, we, we knew this was coming. Hey, you just should have been. We, you know, hey, it's good. We're, we're working on it type stuff where everybody's like somewhat, uh, you know, in, in the same thing as the pandemic, right? Well, you know, who cares if something got done in the lab? We're dealing with the pandemics, they warned us about it, so now we gotta deal with it type thing.

So, uh, it's kind of interesting, but this is just a, a different deal. And, and I think really the, the, the most interesting thing, and, and what I would say we're having the most conversations about is most of the time when we're dealing with ransomware cases, we're dealing with a lot of knowns, right?

We, you, you identify the variant, you know the group behind it, and you kind of know their typical, you know, the typical indicators of compromise, you know, how they act, you know, general what they do, right? In this particular case, there was so much unknown, especially from the beginning, right? Everybody's just trying to figure things out. And you saw pretty, a fairly informal information sharing amongst the community, which was fantastic.

It just makes you wonder what we could do if we, you know, raise the bar there a little bit. But at the same time, people were like, I need to know if my exchange, you know, my exchange server is clean and ready to go. And they're like, well, we don't know. No one knows, right? We don't know what these guys have done yet, what back doors they have planned and what they have done.

And, um, I think everybody on this call, maybe if you don't deal with exchange too much, the number, the, the server that probably produces the most logging and the most crap to sift through is exchange. So it, it was like really the worst of both worlds. You had no idea what we were doing on then. You had to, you had to basically sift through this massive pile of stuff to even start to think about it. So it's a, it's a tough situation. Yeah.

So, you know, as I kind of set the stage here, I started to Chris and then, and, and so what we wanna do is hand it over to Gary, but here's what I was thinking. You know, I ha I spoke with an MSP this week. Not only was every IR firm booked beyond belief, but MSPs that were spending a huge chunk of money, when I say huge meaning the, at an hourly rate with Microsoft's top tier, right? You got a prepay, had a call with Microsoft, uh, 10 days ago, a two hour response, still hadn't heard back.

Yeah. Fortunately, uh, he reached out to me fortunately. And, you know, uh, you know, collectively one of this, this group that was one of the top exchange corporate, you know, MSPs and, and has some of the top exchange talent in the country, actually in the world, that goal, goal known in terms of their Microsoft routes was kind enough to get on with this Ms. P and actually help this big school district, right? Not everybody's that fortunate.

So, Gary, it made me think about, you know, what MSPs need to start thinking about internally when they can't get a hold of a Chris layer. Yeah. That's kind of where I wanna start with Chris. I'm gonna take it in two pieces. The first one is, look what this is revealing both the Microsoft story as well as what's happening, you know, in your firm, Chris, that you know, there's supply and demand, and the supply can't be waiting for this spike like this that happened, right?

And so when you get it, not everyone is gonna be able to get to Chris or someone else in that community or get to Microsoft. So one thing is, the first question, Chris, before we get to what MSP should do, is, I'm guessing that's not a solvable problem. Like you can't stay for your organization for last week, right? Well, I mean, it's kind of interesting, right? We're in the, you know, the Lenten season and, and they say you can't build a church for Easter Sunday.

And that's exactly the, the point here, right? I mean, number one, I mean, we could just talk about the, the shortage of talent and it, and then you even gets worse. And the shortage of a talent and, and security, and then an instant response, I mean, an instant response that to get a forensic person kind of trained up from the ground up, you know, that's a good year, two year journey, right? And so, you're right, Gary, I mean the, the industry as a whole doesn't have today have that capacity.

And even if you could snap your fingers and say, go out and double your capacity tomorrow, it's just not feasible. There's just not enough people out there to do that. And then there's not enough dollars in the world to do that. I mean, the other interesting thing we have with this, and just to say, and I think everybody's seen it, is there's one thing about finding talent, but if you're trying to find talent that says, Hey, can you come work for us?

And by the way, there might be some massive exchange event where you're gonna be working 18 hours a day for two weeks, are you willing to do that? Most of these individuals can go find a job that pays them equal and not have those same kind of, you know, issues or requirements or whatever you wanna call 'em. So it's a, it's an interesting problem. You're right. It's not, there's no easy solution to it. Yeah. So let's take it from this standpoint. Now, getting to the MSP.

So if you are an enterprise company or mid-market enterprise, you know, at this point, there may be a chance that you have some cybersecurity resources on your staff. You may have a contract with an incident response company like you have put some things in place that a big percentage of MSPs don't have in place today, right? Right. And, and we're talking about this when the survey says they only have 25% on average of their customers on prem.

Like, so this, again, Andrew said it could have been worse. So knowing all that, Chris, if you're an MSP, what are some things like you can't go and hire and attract the right people in house, like you're, what are, what can they do to be better prepared in, in the future for this? Yeah, so it's a good question. And It's the question, right? Yeah. Yeah. I've been thinking about it all weekend.

So, you know, let me first start back with just kind of the feedback from some of these, you know, people we've talked to in this particular case, just about their exchange environment and why they have exchange, right? So you have, hey, you know, you have the, the easy one, it's too expensive. We looked at the numbers and anything over 30 people, it didn't make any sense so that we didn't go 365.

You have other people that have said, well, we, you know, we just didn't feel comfortable moving to 365. We wanted to stay with Exchange. We had one that's way up in the middle of Canada know somewhere. And they're like, look, we have terrible internet access. Well, if you have bad internet access, why does it matter about email anyway? And their point is, well, if we have it on-prem, at least we can email each other, which was kind of interesting.

But you have all sorts of different, you know, justifications for why they didn't do 365. And obviously the answer, you know, everybody going to 365 makes sense because for when, whatever reason, we don't know, it wasn't popped like, you know, rest of the exchange environments are. So the, so that's what's interesting about it.

And so then that kind of feeds into the MSP side of things to say, look, you know, we've been talking about customers that may be at a higher risk to you than other customers are. So really, you know, from an exchange perspective, did you look at them having Microsoft Exchange as a higher risk thing? You know, was, you know, there's a, there, there's a potential attack vector that exists there that doesn't exist with your other clients. So therefore, did you really kind of account for that?

Um, did you kind of explain those things to the client when they made that decision not to move to 365? Probably not, because we didn't, not a lot of people would've predicted this would've happened when it happened. Uh, but now you're, it folds into, well, what can you do? And you're right. You know, it's kinda like, um, for example, when I was down here and, and, and was handled the bank, we had contracts for diesel to, for our generators, right?

So bad weather, hurricanes, whatever, you had it. Well, guess what? You could have those contracts with SLAs. But when FEMA comes in and says all the diesel's, ours, there's nothing you can do about it. Right? So again, it kind of comes down to what you're saying is you can have all these great contracts with IRSs, but you know, there's a pecking order. When I was at the bank, we had a premier agreement with Microsoft that put us at the, at the top of everything.

So when we did click called, we knew we got it right. People really don't know about those things. And so you have to be more self prepared about these things, and we learned about that here in central Texas. Can I tell you a quick funny story? Yeah. So when I owned my first MSP, we bought a building, it had seven acres. And, um, at that time we had a data center in the building. And, um, uh, we, we got a, a propane generator, and the generator was three days worth of propane.

And we got a contract that within 24 hours we get more propane. Well, there was an incident that happened that we were down for more than 30 days, unless you said everybody wanted propane, and we didn't get it. And so after it ended, we went and we doubled. We got a seven day propane tank, figured if it goes past seven days, we got other problems and, you know, we never needed it again. Right? Um, but that was just the cost of doing business. That guarantee wasn't worth the paper.

It was, you know, printed on at the time. So, magnify that by all of these things. Yeah. So first thing I heard you say was, take a look across your customers and prioritize them, right? Mm-Hmm. So that when something happens, you're in this spot. So that was the first thing. Um, I, I'll tell you what I was thinking about. I was thinking about the time I've spent with you, Chris, and, and just in recently in talking about IR plans and talking about, you know, the tabletops.

And the first thing I thought of was, okay, if I have customers that have exchange server one, and I don't, and I haven't spent time on ir, do I even know and I can't get support, do I even know what I should and shouldn't say or what I should or shouldn't do? Number one, mm-Hmm.

And two, if I haven't already had these from A-V-C-I-O standpoint, if I hadn't already had these conversations about potentially what could happen and the fact that, you know, we're taking an assumed breach mentality, how different those conversations are. So when you're working with MSPs, do you see those two things? I guess? Yeah.

I I think you could decompose this situation, even though it's it's size and uniquity is you, it came back down to restoration capabilities and preservation capabilities, right? So if you could, if, if you could build or restore, and restore was a tough one. 'cause this thing could go back to early January and some reports are even going back even further than that. So really, do you have the capability to build out a new exchange server and preserve the existing exchange server for forensics?

Then you're gonna be in good shape. Because in this particular situation, you could have built out an exchange server, put all the patches on it and be ready to go. Or the, the other question would be is do you have a plan B to where if exchange is down that they can operate in a different way or different state minimally? And I think that goes a a long way, right?

I mean, we talk about even in ransomware events where email is hit, exchange is hit, you gotta bring up the minimal communication, um, capabilities. And that's what we didn't see, right? We didn't see, look to say, Hey, look, you have a hundred mailboxes, but really there's four or five people that are key. If we can just figure out a way for them to start emailing people, whether that's in G Suite or something, it doesn't matter in the interim, that's better than nothing.

So that preparation wasn't there. Um, and so, I mean, so many people focus and we talk about ad nauseum, but so many people focus on backup and restoration. And in this particular case, that might have not been the right solution, but at least knowing who in the organization needed to have email would've helped you immensely from an MSP perspective, say, okay, good, now I know we need to just build out an exchange server that's gonna take 12 hours. Let's get it done.

I could communicate that to my client. The other thing real quick is we saw, is we saw all sorts of just goofy stuff out there. I mean, we saw people running exchange and QuickBooks on the same server. So that made things kind and that makes you go, go, why the hell would you do that in the first place? And then second of all, why would you have it on a server that's exposed to the outside? And so there's just a bunch of stuff. And so, um, yeah, I mean, again, you're, you're right.

It's, it's, it's, it's having that plan and understanding that an incident response plan is not gonna have every single step together for every single scenario. But you need to know the, the processes, the applications, and especially the people that are the most important. So when you do react, you can react to bring their operations back up at the same time, making sure you preserve things correctly. Okay. I la I only have one more question for you.

It reverts back to the discussion about the tabletop. I, I, I wanna make a, have you make a point on that and sure. But, but you, you know, Chris, from what I, I want everybody to hear what, what you went through there with prioritizing clients, all this forethought around restoration and preservation. But let's go back a step further. Once you have an IR plan, like if you're not changing the conversation, again, we've been using that word, assume breach.

If you're not changing those conversations before the next time this happens, and it, because it's going to happen again, again, you're gonna put more of these relationships. We gotta start educating our customers. We've been, you know, and, and you know, the team here has done, you know, such a great job. But we wanna keep, we wanna keep making sure we're pounding that drum. Okay. The last question I have is, we just went through that tabletop.

So once everybody's safe, is this one of those things? You gotta go back to that tabletop exercise and you gotta take this exploit and you gotta run it through. And, and so you can find some of the changes that are easy for you, you just rattled 'em off. 'cause 'cause you're Chris. But I guess it's different for everybody depending upon where they are. But the, I guess what I'm asking is that tabletop holds up for something like this, right? You can use this Yeah. Definitely holds up.

And it is definitely the, the right time, right? I mean, it's fresh on your client's minds. You can, you can, you have plenty of proof. You know, you have tens of thousands and it's probably well over a hundred thousand. I've been quick not to be critical of people when they just write these articles, these blog posts, and they say thousands.

I'm like, you're not articulating the, the magnitude of this thing, but there's plenty of press and plenty of stuff to support you as an MSP to say, Hey, look, client, maybe you've been ignoring me. Maybe you've been stalling, or maybe we did go through this tabletop, but this is a different situation. Let's spend a few hours and go back through it again and go through this scenario. I mean, it may not be exchanged, but it may be something similar that they have.

Um, you know, a lot of people are fine. They're like, Hey, this is in the cloud. At least we can get to it. We can, we're operational, but what if that was popped? You know? I mean, so those are, yeah. So you, you have plenty of examples to use to run that, run that tabletop again. Yeah.

And if your client doesn't do it, maybe it's worth just your MSP to, to do a, I guess what you would call a kind of a lightning round tabletop exercise with each client and just go through and say, do we know what those big things are for each one of these clients? And let's just go through and see if there's any gotchas for us one by one. Yeah, Gary.

Well, and, and as we go transition over to Ryan, um, I just think it's a, a great segue, Chris and, and Ryan, like, what a better case to talk to your clients about cyber resilience. I mean, right. People don't recognize the, you know, they'll like maybe just downplay the importance of email. It's like, okay, let's run a tabletop, let's shut down your email for two days and see how things go. What do you think? Or a Week or two weeks, Right?

So, Ryan, thoughts on your part, your side Super quickly, Ryan, do you mind if I ask one question of the poll real quick? Yeah, go for it. So, um, or not a poll, but to the audience. So give me a yes or no in chat here has, I'm trying to think of how to word this. Has the events that have unfolded from this Microsoft, uh, exchange stuff caused you to move the needle for all of your clients in a way that you just say, take it or leave it. I'm making changes to what we're doing with you.

And if you won't agree, you're out. Yes or no? Has, has like, has the needle moved for you? The reason I'm asking this question, just gimme a yes or no. The reason I'm asking this question is because you, I talk to many post breach MSPs and that's what they say. They'll say the, the needle has moved for us. And I don't care who you are, I don't care what size you are, I don't if you are going to do business with us, these are certain things that are, that will happen for everybody.

Bar none, no questions asked. Just give, Even if you're an MSP that doesn't have any premise exchange, I think the point Is the same. It this just happens to be this, right? Exactly. Yeah. Okay. We, I had the next question in the poll, just so you know, like if you look at the poll, it was, it was the next question coming, but Okay. And then when the chat calms down, for those of you that said no, give me some reasons on why. No.

'cause I'm also curious, and I'm not saying anyone's right or wrong here. I'm just very curious. So Ryan, thanks for letting me butt in real quick. Yeah, no worries. So I think, yeah, I, I I would actually point out that the exchange server is only the second major email nation state level issue this year, right? The, the original Solar Winds of Ryan one was actually also related to a lot of theft of email.

And so I think, you know, if you're not thinking of email as a critical service and asset, probably need to rethink that real fast regardless of where it sits. Um, but Chris, um, the 80 cases I think brings an interesting scale problem, right?

MSPs most likely can't even fathom 80 incident response cases, let alone the process that needs to go into doing each of those well, and knowing that they don't really have a case management system, they have a PSA, um, how, you know, how do you see them kind of building that, building a robust process?

Um, and, and especially one that involves thinking through kind of all the parties, you know, internal council, external council pr, cyber insurance, incident response forensics, like how, how do they, where do they start kind of building this process to get to the scale of investigation that, that, that you're doing, right? Because whether it's five or 80, at some point you need repeatable process to do this. Well, talk to us a little bit about that.

How do, how do you do it and what do you, what do your recommendations for MSPs? Yeah, So it's a, it's a challenge that, that we've had as well. I mean, there's just, you know, from a maturity level, you know, when we, when I first started seeing really what was going on in the incident response world a few years ago, one of the things that attracted a lot of people to come work for us was the robust tool set.

We had a lot of forensic firms back then were relying upon a bunch of open source tools and just kind of, it was kind of weird the way they were doing things outta spreadsheets or whatever the case may be. And so from an, from an MSP perspective, we had all these MSP tools and they're like, oh, wow, that's, that's some cool stuff.

Not only from a technical remoting in and being able to capture forensics with these types of things more securely and effortlessly than what we were doing with open source tools, but also just from a management of the, of, of the ticketing and everything of that nature. But what I, what I soon learned as things started to increase in volume was that, you know, the, the, the standard MSP tool set, especially the PSA really wasn't, wasn't really well suited.

I mean, I'm sure if you had, you had, you know, if you had the luxury of having some, some guru on staff, they could, you know, sit there and try to figure something out.

But in the end, you're right, Ryan, there's so many parties involved and you guys have that have been on this call, have heard me, you know, since this call has started, how those parties are starting to become more and more because the, you know, just like we've talked about, exfiltration has changed a lot of what we do and the parties are involved. And so now you have people, not only more people involved, but they're even involved earlier in the process. Take forensics as an example.

It used to be a fairly linear type path. You know, we would get in, do business resumption, collect evidence, then hand it over once we were done with what we need to do to forensics. And they would start their work, excuse me. But with exfiltration and with this exchange thing, forensics is needed to be in the very beginning. And so that's a, that's a big change there. Legal needs to be in at the very beginning. We talked about that at length with attorney-client privilege and everything else.

And then you have PR and you have all sorts of other things. And so you're right. So, uh, uh, the tools that we're kind of dealing with today, you know, especially on the MSP side, are not really geared very well to, to be able to supply those needs to those different roles, uh, both internal roles and external roles.

And I would say also, you gotta be thinking about even more so now with what is said and what gets presented to people even more so in an incident situation than you do with, uh, just a simple IT support issue, right? You wanna kind of, you know, it's kind of weird if you, if you, if you have any exposure on the securities side of things, they have, you know, any broker or something on, on that type of thing.

Everything gets kind of reviewed before it gets sent out to a client to make sure disclosure done and things are done correctly. Well, in an incident situations very similar, especially with technical people that aren't necessarily trained to do this every day. I have the luxury, all the people on my staff, this is all they do.

But if you're an MSP and you and your, your, your, your people, you're trained, but they do it maybe once a once every few months or whatever, they may not be kind of programmed as much to make sure that they're careful of what they say. And so the, our tool set today really doesn't, doesn't do that in the MSP world. So we've actually been building out a different platform that really kind of, kind of takes on this, um, this theme or this philosophy of being more collaborative.

But what I would say is a little bit more controlled collaboration would probably be the, the right terminology to use here. Meaning like, you need, I always think of things that to me, like the Domino's Pizza tracker is like the most awesome thing in the world. I think it was it, and what's funny is most people don't know it was built spunk on the back end is what I've always been told that that pizza tracker was kind of built on.

But anyway, my point is, is that pizza Tracker gives everybody that great indication, and now you've seen it replicated, you see it, if you go to Sonic, they do the same type of thing. A lot of places do the same thing, right? So you have a certain audience that just needs to see progress and they want to see progress made. So the tool needs to do that. A ticketing tool really doesn't do that. You don't have time to sift through emails and all that kind of good stuff.

And your client might not even have email up to even be able to get to get to that anyway. And then second of all, you're gonna have legal people that are gonna wanna see a different degree of what's going on. You may have insurance adjusters that are not privy to things that are under attorney-client privilege, but they must be privy to things on their side so they can kind of gauge progress and what money's being spent and all that kind of good stuff. And then you have the technical players.

So you may have something that's all inclusive, you may have just one firm working, or you may have, like, we do a lot and leverage third parties out and about throughout the country or countries, US and Canada to help us out. And each one of those parties may need to have a different view of the data or view status and then have different capabilities to provide feedback or, or progress in that.

And so really the key with incident response, again to kind of decompose it down is that collaboration and just, and then being able to control that collaboration is key. So, so we've been, we've been working with a firm called Exigence that has been, that has a platform to do this, but they're, they're tweaking this platform, uh, obviously for, for our needs.

And, and, and obviously my mindset is because we do so much work with managed service providers is obviously to, to, to make that as MSP friendly, because almost think about this exchange stuff almost, I would say the majority of the ones we deal with have some level of IT provider involved in this exchange thing. We only have a, a few spattering out there that are all just internal it, so in our world, everything's gotta be MSP.

So I think you're right, Ryan, from a, from a tool set, having, you gotta think about things in an incident bubble different than you think about all your other IT stuff and the tool set and the things and how you and the processes around that tool set have to be different. I wish you could say there's a tool out there, it could easily tweak, but I just don't think that's, that's capable, that's possible today. Yeah, I, I completely agree, right?

I think, you know, every MSP has a PSA, every, every MSP has an RMM. Um, those can get you a part of the way, but when you really start getting into these bigger issues where it's definitely a breach, you're definitely have moved from investigation to confirmed breach event. Um, you, you know, you, you kind of get into a different realm of, of the tools you need.

Um, one of the, one of the things I wanted to touch on was, you know, some of Ms Ps on this call are hearing you say, you know, never seen anything like a huge boon in terms of, uh, in terms of work and right. And they may go, Hmm, I want a piece of that. I'm gonna add an S and I'm gonna call myself an MSP and I'm gonna hire myself, uh, someone that claims to know security and I'm gonna start providing IR services. Um, have you seen cases come in from MSPs that were doing IR and mismanaged it?

Like tell us, tell us the downsides of going and like going and adding to your website that you did or MSPs Yeah. Or MSPs, right? Because you know, the MSSP term goes way back, right? And that was really people monitoring your firewall and doing that type of stuff. I mean, we see it now, we won't name 'em, but like we see antivirus vendors quote doing incident response and you're like, that's your antivirus vendor. Why? That's kind of the chicken.

First of all, you got kind of the, again, the thoughts watching the hint house type analogy every time.

So to, and to, in, in my view, just by itself, if you're the person that's responsible for it and you're in turn handling the incident, it's, it's easy picking for somebody to say, well, are you trying to hide, you know, cover your, cover your butt, hide your tracks, all that kind of good stuff in this particular situation, because you could be, you know, you, you could be liable in this particular situation. So I think from an optics perspective, it doesn't look good.

But let's just say even take that aside and say, well, I'm not gonna do IR for any of my clients, but if anybody else calls me up outside of that, I'm gonna do my ir. So I have seen this thing happen all sorts of different ways. Um, I see people say, Hey, we do incident response. And you say, well, what is that? And they're, you know, they're either doing some kind of restoration or they're putting, you know, they're pushing tools out.

There was, there was one that I saw not too long ago where the guy pushed out SentinelOne carbon black and like Malwarebytes all to the workstations at the same time. And like, why? You know, I mean, and it was just killing the environment first of all. And then you get in there and you find out that, hey, he, he pushed these tools out and he walked in and restored their environment, but the stuff that he preserved was not what we needed and it was gone.

And so it, again, you know, I think even somebody mentioned here, Hey, we do this side, we push it to forensics. Well guess what? Today's world, the forensics is so much, so important right up front because there's so much information. I, I can't read enough during the day and get my job done and spend time with my family with all the different things that are coming out with the different, um, indicators of compromise, different methodologies, TTPs, all that kind of good stuff.

So if you think you can just go in there and you've seen you've seen a ransomware incident and you can handle it, it doesn't work that way anymore. They, they're, they're more complicated and it's very easy to make the wrong mistake. I mean, there's so many, Hey, we have our, you know, we have our process in place. We get hit by a ransomware. This is our process. Well, did you even think about your data getting exfiltrated? No.

Or they think about that and they say, well, this is normally how it happens. And we didn't see that. Well, do you know that a lot of these guys do all their work in memory now and you're not gonna see it? Oh no, we didn't know that. Or We're not familiar with this group. We don't even know what they do. I mean, all these things you gotta keep up with. And, and so the forensic, that's all they do all day. I mean, I mean, Wes knows it with the perch guys, right?

Those guys just live and breathe that crap all the time. And so unless you have people that are just, that is that that's what they do, that's their passion, then you're, you're kind of, you know, you're rolling the dice when you're doing that type of work. Yeah. So I'm gonna go in a bit of a different direction here for my, for my last question before I hand it over to Wes. Um, and it's not so much a question, but a statement, and I want you to riff on it with me, right?

There's, there's some, there's a, I have a pit in my stomach when it comes to this exchange vulnerability. And the pit comes from the fact that I think very few MSPs are actually doing the threat hunting work to determine if they were breach or the extent to which they breach rate. And I think it's safe to assume that many of these instances, especially the longer they were up, possibly had multiple threat actors, uh, on those systems.

And, and some of these MSPs likely just were restored from a pre-ex exploit backup, patched it and moved the data back over and we're like, okay, you're good. No more threat actor anymore talk. I mean, do you share that concern and what, what's your recommendation to those MSPs that think I just roll back to 'em and image pre-ex and uh, recover the data and I'm good?

So the challenge is from a rolling back, so number one, you're going off whatever information was readily available to you at that point in time of how far back you can go, right? So, you know, you hear all these January 15th, 16th and it keeps rolling back and you really don't know. And so that's one challenge with rolling back or restoring right in, in other cases we can roll back and then we can scan. 'cause we know exactly what we're looking for here.

We didn't know what we're looking for, right? Right. Um, and so that's, that's, that's the big challenge there. So if that's your kind of planning and you really didn't think it out, or you didn't get some expert to kind of weigh in, then you could just be setting your client right back up again. Now patching seems to be the, the key though. So right. If you patched it good, but you just don't know if those back doors were already there. That's the challenge you have.

So that's where you just say you should just build it back from scratch and not restore. But a lot of people are gonna immediately go to that restore path 'cause they know they got good backups and yada yada yada. And they do that. So again, that's where you gotta kind of put that security hat on and be thinking of it from a security perspective.

And then on the side is, is you gotta, the, the bigger concern here, it really is, Hey, look, well the, you, you had the immediate concern was you gotta get that stuff patched at least, and you gotta get those web shells or whatever cleaned up because you're, you're, you're open for ransomware attack. And that's what everybody's believed and that's what seems to have happened. Uh, to some extent I still think there's more stuff to happen.

Um, but the other side of things in, in this particular case though, is you just don't know what else happened in the environment and that lateral movement. Did it happen or didn't it happen? And if it did happen, have you done all the other preservation things inside that network? So if we do find it happen, and again, back to my point, the forensics right now is a, is is is an, is an evolving thing. It's not one of the things we can go through all the data and say, yep, we're good to go.

We don't know. You're good to go based on all the information that's readily available to us. But later on today, somebody could find a different backdoor or something else in there. I mean, think of SolarWinds Orion, wasn't it just what, two weeks ago? Or somebody else identified something else in there. And so that's kind of a, it's kind of a, a living thing right now.

And so that's the other thing you gotta be thinking about in this particular case is the what if now then you gotta put on the, the hat of risk management to say, okay, really what is the, the probability of that happening? And it, it, that's, that's just kind of a different deal, which I couldn't give anybody a succinct answer on that today. In the end, there were probably so many targets out there that is somewhat like a, a school of fish, right?

There's gonna be some of 'em hit, but the majority of 'em are not. Yeah. Uh, but if you're that one person, what does that mean? And so with some clients, they're like, from an email perspective, they're like, Hey, it's not that big of a deal for us because we have all our, we know all our data's here and all that kind of good stuff. We're not.

But other people, you know, think, think about all that email and all that stuff they have and, and you go back a few years ago to the Sony incident and how much stuff they had in email. And so that's, you know, kind of where you gotta put it on. Yeah. That, that kind of recovery to a good state. Uh, there's another false sense of security there, which is, well this was mass exploitation of vulnerability. They really weren't after me. I was just collateral damage. Right?

So I mean, do you or Wes have any data on like, of the people that actually had web shells, how many of them actually suffered, um, exfiltration? Like is there is, you know, is it, is it high percentage or is it, you know, I-I-I-I-I can't state anything from my side of things specifically.

Um, but you know, and what I'm seeing and what I'm hearing from everybody else kind of, you know, kind of aggregating data if you wanna call it, is is that there, the, the thing about it is it's on the low percentage side of, of any work done more than the, than the web shells, right. You know what I mean? There's, there's the existence of back doors, but where they actually exploited anything done, I think those cases are low.

But what I will tell you though is that you do, there are some, and there is, there's no pattern meaning like, it wasn't like they, they saw the big companies or they saw that they were healthcare, that they saw they were law firms. There doesn't seem to be any pattern to those that were chosen, why they were chosen. And so that's the, that's the deal you have here. You can't just like, you know, Gary mentioned it's not an SMB thing.

You can't just say, because I'm an SMB, they must, I'm not a big fish, so they wouldn't go after me. That's not the case. 'cause we're hearing, um, organizations of different shapes, sizes, uh, a are finding the existence of more activity above and beyond what, you know, the majority are are seen. Okay, cool. If you've heard anything different, Gary, did you have something real quick before we go last? Yeah, just real quick.

I wanna make sure that, I don't wanna shortcut, uh, Wes, uh, on his time because he's in the middle of this. But I just want to, as you're saying this, I'm getting a pit in my stomach because I work with MSPs every day. Okay. And you know, think about the MSP business professional services. It's like its own business. Being good at professional services is its own thing. Then you have to run your tool set. We call that centralized services.

That really has nothing to do with your expertise about how good you are at professional services. You could say the same thing about support desk. Now we have compliance alignment, VCIO, these kind of things. Now we lay on top of that after running five, that's five other businesses we have to be good at. Now you put all these cons, these concerns on top of it, and by the way, I have 10 or less employees. Like it almost feels in some way it almost feels unsolvable.

And that's why I keep going back to at least have your IR plan, at least have those conversations. 'cause we're not gonna be able to be good at all of it. Find the right relationship. So yeah, I, I gotta pit in my stomach too, Ryan And Wes as I hand to you, what I want everybody out there to be thinking about, this really was a wake up call where, you know, we want to do something at least, you know, obviously Wes, Ryan, Gary, you're welcome.

Like Chris and I, bare minimum a webinar on making sure that you guys have a relationship. It doesn't have to be with Chris, but what things do you guys need to be thinking through when an event becomes an incident? Do you get it outta your PSA? We're gonna walk through those types of things. So be thinking of that. I'm gonna email everybody, but we're gonna do a So Wes with you, they all yours, but Yeah, cool. Appreciate that. So, and I wanted to come back to the web shell question on xFi.

So, you know, here's so, so far Perch isn't really seeing any activity there. Um, but one thing we should all be pretty level set on here is exfiltration, depending on the level of complexity and sophistication of the hacker, the bad guy, you may never see it. Um, there are so many ways that I can exfil data out that are like slow that use things like DNS that you'll never normally notice unless you're looking for some certain anomalies, right?

Um, and even still it can be really, really difficult because of TLS if you're not doing inspection on and on and on, right? And so, um, we're not seeing any direct evidence of that yet. But that doesn't mean that it's not gonna happen. And Chris, this probably gets into a question or something you and I were talking about off offline. Um, there's a lot of active web shells out right now and we're even seeing activity from bad guys of like literally fighting over FETs.

And what I mean by that is, one bad guys finding a web shell, it's open. They're actually, um, overstepping and, and there's like this, this fight over FETs of where, where do I have persistence and how can I shut other bad guys out? So I have the predominant persistence. We're seeing that quite a bit. Are you seeing some of that as well? In other words call it a feeding frenzy? Yeah, definitely.

It's, it's this, you know, we've seen the articles talk about somewhat unprecedented, all these groups just jumping on these exploits so quickly, right? Even like if we go to look at some of these firewall exploits that we've seen, whether it's Sonic Wall for net or whatever, you've seen this delay, right? You've just seen it, you know, six months or whatever, even up to a year, and they're finally pouncing on it. It's usually one or two groups at the most.

This one, the scary part of it is you have so many groups in this feeding frenzy jumping on it so very quickly and being very effective at jumping on it. So very quickly. And so you're right. And then, and then you're right just this bat, it's kind of where we start. When I we started off this call, it just seems to be this whole turf battle now and all this kind of stuff. It's almost, uh, kind of funny if you, you know, you really want to date us.

Go back to the, the movie The Warriors and those guys, you know, battling over turf and trying to get back home and all that kind of good stuff. It's almost like you could do a reboot just based on these dgu groups, these attack groups right now. 'cause it's the same deal. They're all kind of battling over the same thing. No one's focusing on healthcare. And these guys say, well, that's your lane. Uh, my lane's gonna be on MSPs and then my lane's gonna be on legal.

They're just, they're just going after everybody and they don't care who the victims are or anything. It's, it's a, it's a, it's an ugly mess. Yeah. It, it, it really is.

And so, you know, our position at Perch is, um, we still, the glass half full is, we still have some time here before we start to like, there's been some indications, like many of you guys probably saw Deer Cry, if you're interested, one of our senior threat analysts, Bryce and met Lock on YouTube and, and I can get the link for you if anyone's interested, did kind of an unpacking of deer cry. We have some, some access to it.

Um, but you know, Chris, our view so far at Perch is, is we're not seeing like the, the doomsday button being pushed quite yet. Um, we're not seeing a level of sophistication or unity around the capabilities of the web shells to push the doomsday button. Although in some cases, I suppose it's theoretically possible, which means that we have some time, um, for remediation at this point. Chris, would you agree with that? Yeah, I definitely agree with that.

I mean, we have been expecting this doomsday to happen just because it, everything seemed to be set up for it, right? Um, and so it just hasn't happened so, so remediation. And I think, you know, I'll, I'll just say this independently, everybody else, I think there's just a lot of great tools out there that are doing a good job at detecting and all that kind of good stuff. And then, and then do provide you the, you know, the information that you need to remediate.

But I think, uh, from a security perspective, um, you know, we always wanna remediate with caution, but I think in certain cases you're gonna need to be, uh, more, I hate to use this term, knee-jerk remediation. Uh, because in this particular case, I mean people, you heard people complain about the patches being slow or slowing the system down and all this kind of crap. And I'm like, well, what the worst thing is about to happen if you don't do anything?

I saw somebody said, Hey, I we're gonna wait a week until my, until the patches, Microsoft's probably gonna come out with some better patches, so we're just gonna wait. It's like, man, you gotta, you gotta, you gotta remediate quick. And if you're kind of sitting around and you're encountering issues and you're not reaching out for help, and if you're waiting on Microsoft to, to tell you what to do or to respond to your call, you're, you're setting yourself up for an attack. Yeah.

You know, in every surgeon's tool bag, I'm sure there's a scalpel and there's a bone solve, right? Uh, and there are times that we use one for one purpose and, and another for another. And probably a telltale sign of immaturity around incident response is the fact that you're not even enabled to have the bone saw. Right? And when do I even use that? And it can be a messy, um, kind of work, man. What a grim analogy I'm using.

But, um, the, the truth is that sometimes incident response requires some drastic action very quickly, right? That's right. I mean, that's why I have the, I put the background today, it's the, it's the battle. It's a different, when you're in, in the battle, there's different decisions to be made and, and different responses to be made. And this is exactly one of those situations. I mean, if you don't have any other choice but to take exchange offline for 24 hours, you gotta do it.

Um, I mean, and you, you better be able to explain to your client why you, why you have to do that. And if you're unable to do so, you know, that's a different problem altogether. But try to explain them Before an incident if you can. Yeah, exactly. Try to prep 'em. You're exactly right. And so, you know, it's the same thing is, you know, you, you get in a car crap, pretty bad car crash, what's the first thing you wanna do is, you know, disconnect the battery.

You know, you don't want anything catching on fire, so you don't even think about what is gonna damage your presets on your radio or it's gonna blow some fuses. You don't think about any of that stuff. You, you want to stop the car from exploding. It's the same situation here. You're gonna have to the pull, the plug, cut the power. You're gonna have to do those things in certain situations. And this is, this one's the perfect example. Yes, indeed.

So Chris, I wanna pull away from the exchange stuff for a while. I mean, I know we, we talk about that ad nauseum and, and we should, it's important. But can we just, I'm gonna let you pick, uh, you know that I like to do this with you a lot, especially when you and I are just like chatting, uh, offline. Pick an incident response, pick an ir, um, scenario you've been in recently, especially with an MSP and just talk about it. Let's kind of just chat about it a little bit.

Just pick something that has been interesting, um, that's thought provoking, that he would have some lessons learned for all of us in it. I'll let you pick anything that you'd like. Well, I, I, I'll, I'll still say this one 'cause we still see it quite a bit and when we've actually come across it again, is, um, and I'll just harp on it, is this recover with the, you know, hey, we can recover. We don't need to deal with the ransomware at all.

And then you come back two, three weeks later and their data is out there on the dark web and, um, that's, that's the big issue. Um, because okay, look, they're back up and operational, but the legal side of things just looks terrible, right? I mean, you, you probably did everything you shouldn't have done from a legal perspective in that particular situation.

So that's really, you know, we haven't to be fortunate, I guess, and I'm sure they occur out there, we haven't come across too many where the MSP has been the target and has been the one that has been, you know, all their clients have been hit. We've been fortunate not to have to deal with those. And maybe that's, that's just a good sign of the M MSP community as a whole.

But these ones about, I think just again, you know, this common theme is MSP's just rolling with it, thinking they got the things covered and they wanna get their, and, and I, to me they think they, I mean, you wanna give 'em the benefit of the doubt in some cases, but you're like, look, I think that yes, restoring to their customer very quickly and getting them back up into operational seems like the most logical decision. It's not, but it seems that way.

But to me, I think really the MSP is trying to sweep stuff under the rug. I really do. Because, um, because if it looks any worse than that exfiltration or whatever, it's gonna make the MSP look bad. So I think that's what, what, what's happening in today's world, to be honest with you. And if you're not sweeping it under the rug, so be it.

But the optics are gonna look like if I Don't know Chris, I mean, I, I think that would be optimistic that that And given too much Weight, there's much force into it. I don't even know that there is. Yeah. Like they've even gotten to that point in some conversations I have. They're just doing the same thing they've always done, which is the client's down.

We need to get 'em back up and doing everything that was right is now a hundred percent wrong in this environment, as you've explained to people over and done a great job, you know, explaining to our, to our viewers And the dollars are so high. You know, I had somebody call me the other day and it's like, Hey, I had this, you know, I had this one. Can you just check the price for me? I mean, we recovered everything. They just checked the price.

So I checked the price and it's like 3 million bucks. And I'm like, that is not what, I'm like, well, that's the price man. I don't set it right. I mean, it's not my deal, you know, and so you kind of, you say, look, we're able to restore, but then this whole data exfiltration goes on and you want to try to figure out some stuff. And sometimes that data can be very damaging.

And so you, even though you've recovered, there may be some reason to pay the ransom even though, you know, it's usually, most of the time if you're recovery, you still don't wanna pay bad guys. But sometimes if that stuff gets published, it can be personally embarrassing to the company or to their clients. And so you have to do that. Go ahead, Andrew.

Yeah, I Was just gonna say rhetorically again for you and West to maybe just chat about, you know, there's been so much chatter about CMMC and C, you know, controlled unclassified information. Well, this has got, talk about exfiltration and what that might look like. I mean, right West, I mean, and Chris, I mean that Yeah, I'll let, I'll let Wes chime in. Um, I am still, we have dealt with some situations where the data that was exfiltrated, you just gotta go, holy crap.

I mean, you were securing that or how, and they have an MSP involved and I just, you back there, just no one's talking about it. And it's the stuff that people keep and they keep it forever and they don't need to keep it. And the stuff that's out there, and it's, and these guys, these, these attackers, they're using tools. It used to be, it was a pain the butt for these guys to get, you know, a couple of gigs of data.

Well now, 5, 6, 700 gigs, one, two terabytes of data they're sucking and they're getting it all. You know, we, No, you're right. You know what's funny about this, Chris, you remember about three years ago when we first kind of got together and started chatting about all the cybers, and you and I would talk a lot about data retention policies, but we talked about it from the frame of reference of like subpoenas, depositions, things like that. And you must provide whatever you have.

And I come from a banking background where, you know, data retention is extremely, and Chris, you do too. It's extremely well-defined and banks typically like to dump that data the instant they're able to, because banks are usually party to lots of lawsuits, even though we're not su we're not in there, we're we're called in. Right? What do you have on this nasty divorce proceeding of X, Y, Z? Right? Whatever it may be.

Uh, and those are the things where if the bank has the data, even if it's past retention, we, we must supply it. And there's a corollary here to cybersecurity in terms of incident response, the same thing. We can actually do ourselves a lot of good by nailing retention policies as much as the client may get mad and say, well, I can't find that date anymore. You deleted it. Yeah. 'cause it was after that.

But again, it comes back to that education of why we delete, it's not just for the lawsuits, but it's also for this age, this modern age of where standardization around data exfiltration happens from bad guys, right? Yeah.

And I, I think the standardization for exfiltration, I think, you know, the other side of things is is, I mean, you just look, I mean, in today's world, everybody's ready to pounce on anybody about any sentence they said or any two words they use in the wrong order or whatever. I mean, that's reality. I mean, it sucks. I hate it. I'll talk about it all day long, about this whole cancel cu cancel culture and everything like this.

I mean, I mean, whether you set it on camera or you set it on email or whatever the case may be, is it, it it's gonna get you. And so it just, there's just, there's too many overwhelming reasons why to get rid of data. It just, people just don't wanna do it. And it would, it would solve a lot of these issues. I mean, just think about this exchange issue. If, if, if you just didn't have so much stuff in exchange, then would it be that harmful to you to, to lose it?

You know, We're definitely just recording next week. Yeah. I mean, there's a reason Sunil is talking about the DIE triad, right? Yeah, Yeah, for sure. Yes. Um, and, and this is why I have no recollection of these, uh, facts, Senator, I, I have a, I have no aspirations to be the host of the Bachelor or Bachelorette, so I'm not, um, I'm not concerned with what I say. All right, Chris, thank you so much for your insights. Um, I'm gonna turn it in a couple minutes. We have left back.

I'm Andrew, I'm gonna give it back to you. Yeah, absolutely. Well, Chris, fantastic job as always and thanks for coming on. I, um, like I said, I mean, I, I think we're at this the point where we really need to do something more for the community. I, we talk about, you know, we just got through the cyber resilience workshop. So, um, you know, Chris, any thoughts on and closing thoughts on what we might be able to do in an upcoming workshop?

And, you know, I know I'm putting you on the spot here, but what, what might we be able to do with the group and, and kind of walk through those scenarios? Yeah, I think maybe, uh, I kind of thought about it while I was talking on this, but maybe we come up with some, you know, maybe, uh, accelerated table. Maybe we don't actually do it, but we kind of, kind of come up with some accelerated tabletop exercise you can do through all your clients.

Maybe you do that once a month or once a quarter or, or you base it on the criticality, but maybe we could come up with some new type of incident response risk assessment blended thing to help the MSP figure out who goes first, second, and third in the pecking order, just like Microsoft's been doing in this event. Right. Right, right, right. Okay. Well, awesome, Gary, closing thoughts? Yeah. Um, I got three quick things, but first I, I wanna say, um, listen, this was a great call today.

Everyone who's a a listener, anyone you know who's in the MSP business, um, please get 'em to start and listen to this. Uh, and I don't say that because we don't have a financial, there's, there's no, we don't make money from the cyber call. We don't. Why am I, I'm out, but we have to, but we have to get, get the word out. And I don't know there's a better place than the people that are on here. Three quick things. One personal thank you to Chris.

Chris, I get a lot of people, my customer's contacting me when there's an issue, we've send them over to you and you're so helpful, so personal, thanks from true methods that you've helped so many, you're, you're an ACE two VCIO assume breach internet response. Okay? They're the three things that everybody needs to be thinking about. And number three, everyone who's can hear my voice, you're not charging enough, You're Not charging enough, and you're assuming too much risk.

We're not, you're not sharing it with your customers, and it's Not, not your, it's not all your risk. Yeah. Yeah. Great, great points. Ryan, any, any closing thoughts from you? No, I mean, I think this, you know, if you weren't convinced three months ago that we were living in the assumed breach era, like, I don't know what evidence you're gonna need at this point. Um, but I think the extra learning from this, hello? Anybody there? Bueller, right?

The extra learning from this event is, it's not enough just to assume breach. You need to be prepared to respond at scale in your customer base. And that's a different ball game than even what we've been talking about in terms of getting your house in order. And so, you know, it, it, these, these, they're, these hits are just gonna keep coming, right? And so if you're not taking this seriously yet, like now is really, really the time to start getting your house in order. Yeah.

Wes, I always come to you at the bewitching hour of two. Don't give, we're done. I got Something to close this on. Uh, hey Chris, sorry for down voting you yesterday or last week. I'll come back to haunt you. Maybe I'm not sorry. That's all I got is coming soon. Awesome. All right, everybody, have a fantastic week. We'll see you next Monday. Thanks guys. Thanks Chris.