Hear from 1st Responders at Kaseya, Blackpoint & Solis Security

All right. Welcome everybody. We are live. I'm gonna set the stage quickly here because we have a lot of people coming on, and of course, uh, Gary, what else? But I lost power at 6:00 AM so my WeWork is a Starbucks. Um, I'll do my best folks, and, and please Go with Andrew. Listen, before we, any thoughts? I, I know it's been a long weekend for everybody.

We're gonna talk about it, but can we still say, Hey, we gotta stick with, we gotta, we gotta stick with our attitude and we gotta stick with things. So, uh, let's go ahead. I'll let you, uh, take away and introduce Mike. Okay. Let me share just real quick for everybody the format, and I'm gonna request patience for everybody. First, thank you all for joining us. As it, you know, impromptu as it is, I wanna thank Mike Sanders from Kaseya for coming on and speaking with everybody.

I want to thank Huntress, uh, for their support. We're gonna be hearing from John Hammond, um, and, uh, black Point CEO, John Murson. Um, we, I'm hoping still have an MSP coming on towards the latter portion as well. So just realize folks, um, where a normal cyber call that we do on Monday, like a routine is, you know, clockwork and easy. Um, Crowdcast has a six window limitation, and so therefore, um, like, I'm gonna be rotating people on and off.

So we're gonna start off with Mike, uh, Mike Sanders, VP of Kaseya, and, um, Mike, uh, the floor is yours. I appreciate you. I know, um, it's been an incredibly stressful time, and, uh, I appreciate you coming out and addressing, um, the folks here on the cyber call. So, with that, um, please take it from here, and then I'll have Wes, Chris, uh, and Ryan chat with you a little bit, uh, for the next 10 or 15 minutes. So thanks again, Mike. Sounds, sounds good. Yeah, definitely.

Uh, listen, I appreciate, uh, the, the chance to come out here and, um, you know, obviously I think it's pretty, pretty well known by now that, uh, that, that we've had a, a bit of a, of a, you know, a rough go of it for the past few days here, uh, dealing with, with this security event.

Um, for, for anybody left in the world who doesn't know, um, you know, we, we had a, a pretty, uh, a pretty large, um, kind of outpouring of, of support and everything else from the MSP community, which I, I really appreciate and, and really think it was, uh, awesome. The way that most people have, uh, kind of responded to us over this. But, um, for anybody who is not aware, we had a, uh, a security event that happened on, on Friday.

And, uh, and it was, you know, something that, uh, we started to get just a little bit of, uh, of noise, uh, kind of in the late morning. And then, um, it kept growing just gradually throughout, kind of, you know, till around 12 o'clock. And then, then we got a cluster of, of, of calls, we said, you know what? We gotta more room.

This got together, uh, and, uh, and, and by, you know, two o'clock, somewhere in the two o'clock range, you know, varying, depending on what, uh, what, uh, kind of segment of our, uh, of our, uh, customer base it is. Uh, we had, uh, we had shut everything down and or instructed our, our MSPs and, and, uh, and on-premise customers to, uh, to shut everything down.

Um, what, you know, we, we spent since that time, uh, a hundred percent of our time, uh, working on kind of the, the, how do we get our clients back up? What was the cause of this, uh, as, as it's been kind of put out there already, we're working very closely, uh, with FireEye, uh, on the incident response in particular. Um, they've been, you know, fantastic, uh, for us. Uh, and, uh, and that was a decision that we made, you know, pretty much immediately to, uh, to, to work together with them.

But what we know today, uh, and again, there's, there's, you know, there's gonna be a lot more, uh, depth to this as, as time rolls on, uh, and, and we can be more, uh, open about a lot of these things. But, um, what we know today is that, you know, we did have a zero day exploit. Uh, we did get, uh, uh, you know, some, uh, of our servers impacted, uh, as a result of that.

Um, you know, and, uh, and they, as a result of that attack, uh, were then leveraged to, uh, deliver payloads into, uh, some of their client base. And that, that's, that's kind of, you know, I think what, uh, what, uh, that attack vector, uh, looked like. You know, there's been a couple misconceptions out there. I think the big one is, and, and I just, just so that everybody's clear, I've been taking point on the, on the, uh, on the incident repo response around the forensic side of it.

So I, I've got, you know, I'm not talking through, you're not talking to a media relations person or somebody like that, that's trying to, you know, give you something that's, uh, secondhand. I, you know, directly from what we have seen, this is not what we would define as a supply chain attack. This was a, a zero day exploit that, you know, somebody was able to take advantage of and, and was able to, uh, to, to get access to the, uh, Kaseya VSAs, uh, as a result of that.

So that, that's something that, you know, is, is really important because it's, you know, it's being, uh, characterized as, as something that it's not.

And it's, uh, you know, we, we've definitely, uh, wanna make sure that everybody knows that to this point in time, neither the FBI, you know, department of Homeland Security, FireEye, none of us, uh, have have any indication that anybody has made it inside the Kaseya network or has any access to Kaseya employees or Kaseya data, or, or certainly our, our code base. And, uh, and so I wanna be very clear about that.

Um, and, you know, another thing that, uh, that keeps kind of coming up, there's been a lot of different, uh, there's been a lot of speculation on a number of people impacted. Um, and, you know, we've got a hotline out there. We've got, uh, a particular email address for people to report it. Uh, I've gone on Discord, I think is some of the people in the, in the chat here, uh, probably know, uh, and ask for people to reach out to us.

We are still below 60, uh, actual impacted, I don't wanna say impacted. 'cause obviously, you know, we had to shut our, our services down. And, and I understand that that impacts everybody, but directly impacted customers. We're still, uh, not seeing, uh, a a large amount of those. So, um, for, you know, for that side of it. Um, and, you know, I think that's, that's one side, you know, one, one thing that we wanna be very clear about.

Um, the other thing that we want to be clear about is, you know, the number of customers that we're impacted. On the other side of that, we don't track how many customers our MSPs service, right? You could have a hundred of our agents across a hundred companies. You could have a hundred of our agents at one company. We wouldn't, you know, we wouldn't know. Uh, it's not something that we would track. We, we don't, you know, snoop in the data, uh, of our MSPs.

And so when we get asked how many, you know, end customers of the MSPs got impacted, um, that is a, uh, you know, that is something that, uh, we don't really have an answer for by our estimates. And, and, you know, I would expect that this is actually a very high number, but by our, our estimates, um, it's, it's, you know, below 1500 total customers.

We'll learn more about that as time goes on, and we'll be transparent through that process, you know, but we, we, obviously, it's tough for us 'cause we just don't have a way to tell you, Hey, this is exact, you know, exactly how many, because, uh, our, our technology doesn't work that way, and we don't, we don't keep records of, of that. So those are some of the things.

Um, you know, I I think, again, from overall, the response to this from our client base, from the, from the people that I've spoken to, has been very, uh, very positive given the circumstances, right? I mean, obviously nobody wants this to happen. Um, obviously this happening to one client is, uh, is is too many. And, uh, and, and, you know, we're, we're definitely taking it very seriously.

But, um, the flip side of that is, you know, again, I think we're very, uh, you know, not, I don't wanna say proud, that's the wrong word, but I think we, we did as good of a job and as fast of a job as reacting to this as we possibly could, the timing for us to say, Hey, we've got a problem. Uh, take everything down and notify all of our customers. I think we saved a lot of people from getting hit a lot of people. And, uh, and I think we did that.

You know, we, we, we had a plan in place that told us kind of what that needed to look like and what the timing and what we would, you know, what steps we would take. But we didn't hedge on it. We didn't delay. We went, you know, straight into our plan of action, which included the communications and shutting everything down.

So I think, uh, I think that's, you know, something that's, that's important, important to, uh, to note and, uh, and something that, you know, we'll go back and look at and, and, uh, and make sure that, you know, if there's ways we could have done it even better, that, that, you know, we'll address that. But I think we did a, uh, a pretty, pretty amazing job on shutting down as quickly as we possibly could and, and get the message out.

Um, I know there were a lot of people who helped get the message out in the community as well, and that's, you know, obviously awesome. I think we saw a lot of that, especially on Discord and Reddit, things like that. And that, I think probably helped a lot of people as well. Um, so that's kind of the, the, the 10,000 foot view, uh, the, where we go from here.

Um, you know, we've got, uh, a plan, uh, with, uh, restoration of services that frankly, I think will make us the most secure technology, uh, out there from an RMM perspective. We're not just going and, and flipping a bit from one to zero and saying, Hey, it's fixed. We're actually taking a multi-layered approach. We're working with, uh, with FireEye. We're working with, uh, you know, uh, much more aggressive WAFs than, uh, than than we have in the past.

And we're gonna be doing something that is, uh, multilayered and is going to, uh, protect our clients in, in a way that I don't think we've ever seen, uh, in this particular, uh, space with these, with these types of products. So, um, you know, hopefully that is, uh, that is something that, uh, we can get done very quickly.

But that process, which, you know, we've been working on nonstop since, since this event took place, is the reason that it's not something we can just patch in one minute, right? If it was just a, Hey, we're gonna go get rid of this, you know, thing in this page, in this whatever, um, or in this configuration file or whatever, that would be a lot easier. Uh, and we'd be done already.

We're looking to do something that is much broader, and we're looking to do something that's much broader that affects, uh, not only our, uh, on-premise customers, but, uh, but also the cloud, uh, infrastructure. So something that keeps everybody, uh, secure across both of those. So that's, that's kind of the high level view. That's, uh, where we're going. And we will put more details out on that as the day unfolds.

Uh, you know, and we, we track closer and closer to actually getting, uh, everything back up and running. The one thing that we would, uh, you know, really like to do more of, and, and I think it's been kind of a, a weird thing for us. Um, you know, we have, uh, had, uh, some vendors, uh, out there that, uh, that we work with that, um, have done things like cutoff, uh, integrations and things like that to ancillary products from the Casea VSA.

And for us, that's, you know, we, we would love an opportunity to kind of understand that process. Um, you know, it's not something that is, uh, is, is a logical decision. Uh, and frankly, it's not something that, uh, I haven't had a conversation with any of the vendors that have made decisions like that, you know, notably, uh, ConnectWise with the, with the it glue stuff to take them through what we know so far, so that they could make an informed decision on it.

And, uh, you know, for us, that that was a little strange given kind of our reaction to a similar breach of their technology, uh, in our product, uh, a few years ago. So, you know, definitely, uh, you know, if there's anything that I could say on a broader scale for us, uh, the reaction from our clients has been overwhelming. Uh, it's been overwhelmingly positive, given the circumstances.

Obviously, nobody's happy about it, but, uh, but they've been very understanding and they, they've been appreciative of our communication levels and the way that we, uh, the way that we handled it from a, uh, from a shutting down perspective, um, you know, and, and we're just looking to, uh, to kind of figure out how we can bring this, uh, this system back up as quickly as possible, but while really, uh, helping out the, you know, kind of, or not really helping out, but really ensuring that, uh, that we've taken every possible precaution to make sure that no other attacks can occur.

Mike, thanks for, again, thanks for coming on. Sure. And, um, 'cause I know this isn't easy, um, and, uh, um, I appreciate what, you know, you being, you know, sharing some transparency here with everybody. Um, I'm gonna open it up to the floor, um, to Ryan, Wes, and or Chris, um, thoughts, things that you want to kind of chat with Mike A. Little bit about. And, um, so Wes, you see you shaking your head. Anything you wanna start off with, bud?

Yeah, I'll just start off and say, Mike, first, thanks for joining us. Um, it's certainly not an easy time for anybody, let alone Kase and, um, wanna appreciate, say thank you for joining. Thanks for the, the updates and what's going on. Um, I think largely, um, I, I guess maybe one question for you kind of out of the gate is, um, where do you see the next week kind of leading for you guys? Do you sort of see, are you guys in the response and recovery phase right now?

Where do you guys see the next week unfolding for you? Yeah, I think, uh, you know, for us, we're, we're looking to bring our, our cloud, uh, customers up first. It's, it's a more controlled environment. It's, uh, it's easier if we have to pivot, uh, on any of the things that, uh, that we've implemented, most notably, kind of the last stuff.

And the, the, you know, the backend, uh, uh, SaaS SOC offering, uh, that, that we're applying across it with, uh, with kind of some of the intrusion detection stuff, you know, that's, that's new for us. And, and trying to kind of put that out in the wild of, of, on-prem, um, makes it harder for us to kind of tweak it, tune it, get it right. Uh, and so we, we are going to be doing some of that, uh, in, in parallel.

But we are, you know, we are definitely looking to bring our, our cloud, uh, customers up, you know, today. And, and then we would like to bring our on-prem customers on, you know, over the, the following 24 hours after that. So we're not talking about weeks or months or something silly, obviously it would never be, uh, that long, but it's not even, you know, we're not even anticipating it being a long tail.

We just wanna make sure that it's, you know, that it's, we're not worried about it from a security perspective. I think we've got that buttoned up. You know, there's, there's a, a, you know, a tremendous amount of, of work that's been done around that. It's just a, Hey, how, how's the performance gonna look? Will there be false positives, uh, you know, things like that that we're gonna need to deal, deal with? And it's a lot easier to do it in an environment that we control. Thanks for that.

Mike. Wes, I'm sorry, Ryan, anything? Yeah, I think there's a, I'm, I'm mainly just looking at the questions MSPs are asking. I'm gonna, you know, ask them as proxy. There's, there seems to be a theme in the questions around, um, when the vulnerability was known, um, you know, when you were, you know, what the remediation timeline was, what application security practices you have.

I know there's some information out there from the Dutch cert about, um, responsible disclosure they were working through with you, and they've come out and said you were engaging in a responsible way, you were communicating with them. Then, you know, it seems like maybe just poor timing, but could you maybe walk MSPs through a little bit more about Yeah, Absolutely. That CBE and your process and, and Sure. The timing and, and just help them get a sense for how that unfolded. Yeah.

I think, uh, I think that, you know, it's a great question. We, we've been working with, um, with them for, for some time here. Um, and, and, you know, they've been discovering a, a bunch of, uh, uh, of, of, you know, lower priority and, and some, uh, higher priority things that we needed to fix. And so we, we kind of put it all into, uh, into our, our development cycles.

Um, we had actually done two, uh, patch releases already taking into account different things that, uh, that we had discovered together. And I think, you know, if you look at the way that they, uh, came out and talked about it, we've been a, you know, a good partner working through that with them. They've been a good partner in helping us, uh, discover things. And it's been a, it's been a good, uh, kind of process. And, and frankly, yeah, I mean, we, we, we got the timing on.

It's just unbelievable. I mean, it just, it's just, you know, really unfortunate that, um, we were so close to, um, getting some of the things. Now that being said, you know, we're, we're still, I don't wanna say that, uh, that it's impossible that, uh, that, you know, the things that they discovered, if we, if we had patched, um, the last remaining things, you know, a day earlier before this, uh, this, this, this attack, that this attack couldn't have happened.

I, I don't wanna speculate on that. I will wait for guys like FireEye to go through, um, you know, the, the full, uh, range of, of things. But, um, either way, you know, it's, it's, it's something that, um, you know, we take very seriously. I, I think people are, you know, they're, they're putting it out there like, Hey, you guys are working with these guys for so long and you still got hit.

Well, I think what's really important here is we were already working with people on these, these things before we got hit. Like, this was a proactive thing for our company, and it's something that we take very seriously. Um, and, you know, we, uh, we will continue to do that moving forward. There's, there's absolutely no doubt. Oh, and one other thing on the timeline there, there's really no confusion from our side on the timeline at all.

I mean, we, we got a few, you know, a few few hits earlier, like I said earlier, sorry, later in the morning on, uh, on, on Friday, um, at Eastern Time, um, by two o'clock, you know, by sometime in the two o'clock range, we had shut everybody down, um, or notified everybody shut down. Yeah. By the timeline comment was more around the, the timeline between when you became aware of the CBE and when you were fixing it.

But, um, you know, again, going back to what the Dutch cert said, they said you were working on it. Mm-Hmm. You were engaged with engaged with them in a responsible way. Um, and so I think, you know, that is important for MSPs to know that Absolutely, yes, there was a vulnerability. Yes, it existed in the product.

Yes, it was exploited, but it wasn't, you know, I, I think it's dangerous for there to be claims that there was negligence here, because what the Dutch sir has come out and said is that you were working with them in a responsible way and, and assert for a a, a country does not come out and say something like that Exactly. If they're not actually seeing it. So, you know, are there process improvements, I'm guessing? Absolutely.

And you're gonna be having a lot of those conversations over the next few weeks, but, um, you know, just wanna make sure that people know that that's, um, Yeah. And we'll, we'll have, we'll have more detail on that, uh, as, as time goes on, but I, I really appreciate that distinction.

I think it is something that, you know, it, it is something that we, uh, we appreciated them coming out and stating that, um, we've had the same response from, from the FBI and csun, the department, uh, of Homeland Security in the conversations that I've been on with them.

Um, and, you know, again, it, it is one of those things, it's not an if it's, it's a, when you gotta do everything you can, everything in, in your company's power to, uh, to, to protect yourselves and, and your clients from it.

Um, but on the flip side, you better have a really good process and a plan for, uh, mitigating and getting back, uh, into, uh, into, uh, a stable state and getting everybody back online, because there is no, there's no, there's nothing out there that is, uh, is a hundred percent never gonna have something like this happen. It's just, I wouldn't, I wouldn't bet on it. Yeah. Well said, Mike.

And, uh, Ryan, thanks for kind of pulling that together and monitoring, you know, the, the theme of the audience. Chris, did you wanna, uh, no pressure. I know you're no overing and just being, you know, uh, do, do you have anything or should we bring on John Hammond? You tell detail?

Chris, I've seen some of the questions too, and I know people are gonna have a lot of, uh, questions afterwards and, and, and knowing Mike and Kase, they're gonna, they're gonna come out with some post-mortem stuff in more detail later on. And so I, I appreciate everything they've done. The other thing I would say is, yeah, I know a lot of people are gonna have some, you know, I'm gonna call 'em Monday morning quarterback questions. And I appreciate those, uh, immensely.

And I think I just give those some times, you know, there's a lot of what if scenarios, and I deal with that all the time. And in cases where we get third parties involved and they're like, well, how do you know this didn't happen? And what if, what I would just say, let those things happen. I know we want a lot of answers immediately. Uh, and again, I think Kade did their part. Um, everybody's gonna find this opportunity to improve upon, and, and we're all gonna continue to work together.

And it's, it's, uh, it's kind of where I'm at right now as far as this goes. Yeah. The only thing I I I'll add is, um, well, Mike, definitely thanks for coming on today. Uh, and I also, I want to thank, uh, Kaseya from my standpoint, you know, they made his decision to leave me out of the comms team, out of the response team. They understand my role in the industry, uh, as being independent.

So they purposely didn't give me access to any information, uh, so that I could come here today and, and, and play my role and, and as I do in my thought leadership. So, uh, I appreciate that, Mike. Definitely. Yeah. So, um, Mike, big thank you, Mike. I'm, this is recorded. Um, and I'll send the link to Dana and your team. Sure. There's a bunch of questions. I wouldn't put you on the spot and ask those, 'cause I know your team needs to vet them and look at them. Sure.

But, um, I will make sure Dana sees all of those, and your comms team can be aware of them and respond, uh, in turn, you know, maybe in a blog or whatever, what they can and cannot answer in a timely manner. Yeah, absolutely. And I think, you know, the big thing for us, I, I wanna be real clear here.

The, the biggest thing for us is, is helping all of our clients, the ones that were directly impacted, the ones that weren't, get back online and get everything back on track, we're heads down on that. So if, if it takes us a little bit longer to, uh, to respond to, to some things, you know, it's, it's not for a lack of, uh, desire, but, you know, there's some things that we don't have the answers on, and they're not a priority for us yet.

There will be plenty of time for us to dig into it and, and learn lots of details on it. Um, but we will be as transparent as, as we can without risking, uh, you know, further attacks. We wanna be clear that, you know, we, we don't wanna open up any, uh, additional risk for, for us or for our clients, uh, through, through some of the conversations. But we're certainly trying to be as transparent as we can.

And we're certainly not trying to be, uh, you know, uh, anything except, uh, upfront with, with everybody as we go through the process. Well, Mike, again, no one, uh, one again, big thank you, big thank you to Kase, to you. A huge amount of empathy goes your way. No, no one wants to be in your seat. Um, and so, uh, I appreciate the community coming together as always. I, I can, I can assure you, you know, you, you know me. I, I know I've known Gary for forever.

You know, we feel significantly worse about the impact to, to our clients, uh, those that were directly affected and, and the rest of it, that feeling sorry for ourselves. Uh, and we're just, we're just gonna work our butts off till we get it back up and running and, and get it, you know, as secure as we can. Yeah. Mike, you and I have known each other since 2 0 4 starting this industry, so I appreciate, uh, you know, your commitment to it. So, Absolutely.

Uh, I'm gonna push you off onto the, uh, audience and bring up John Hammond. Thanks. You guys have a good one. Alright, Bye. Alright, Lemme get Mr. Hammond here. So, while we're bringing up, uh, John, um, maybe one thing I would just say that I'm thinking it's going through my mind is, you know, uh, wow, think about this in 2021. This is so much better than we would've handled this four years ago. Don't you guys see the same thing?

Um, I, I've noticed a much better communication across the board. I've noticed, um, our capabilities of information sharing are better across the board. Um, you know, even in reality that's sunk in, like these things happen and they're horrible situations, but they do happen. Um, so, you know, yeah. A huge shout out to, to Michael For sure. I agree. Yeah.

The other thing I wanted say is, there's a, there's a lot of really good questions coming in from MSPs, valid questions about, you know, why, why was it fixed in the cloud and not on-prem, and how long were you working on it from when you known it to when you fixed it? Like, all of these are important questions, and they are questions that I say is going to have to answer. However, they're not questions that they are going to answer today, right?

They are still in the middle of responding to this, restoring customers regaining competence in the integrity of the environment, figuring out how to release this patch to all the on-prem customers, getting them back up and running. They are still very deep in the middle of this. And the questions that we're asking are, are kind of retrospective postmortem, right? They're the types of things you do in an incident response phase once you have recovered.

And so you are valid in asking these questions and wanting these answers. You just, I, I would ask that you understand where they are in the lifecycle of their response and that you're just asking these questions maybe a little bit too early in their process. But you should definitely be asking these questions and seeking your, their answers in the coming.

And they'll come And they will, they'll come on to, you know, uh, future, uh, cyber call, when it it's appropriate to answer all of those questions. They permit it to come back. Yeah. I know there's some questions and there, and there's some questions out to me. So after John's done, when he gets around to me, I'll, I'll, I'll give you my opinion on some of those things as well. But you're good. Good. Thanks. That's the exact messaging, Ryan, that needed to go out. Thanks.

Thanks Ryan for that, as always. Okay, John. Hey gentlemen, maybe, uh, what, what, uh, let's see, Friday to Monday, what are you averaging about? Uh, four hours in the last four days. Um, maybe, yeah, Three to four hours of sleep whenever we can squeeze it in. I, I like seeing John better when we're doing simulations for, for our, for our audience. Not the real thing. Well, thank Good you can't smell some of us on this call, 'cause we probably wouldn't smell At this point.

Well, Chris, you were here in, uh, Tampa with me over the weekend, and then lemme tell you now. So, so John, I I picture you doing one of your, uh, you know, normal videos, uh, your CTF stuff with your big monster, uh, several monsters in front of you. Um, I'm sure that is the case, so I know the folks out there wanna hear from you. So let me, let just let you set the stage. I'll set the stage for you, John, and you just riff where you need to.

And again, I'll let the team chat with you after, but maybe take us through the chronology. You guys were early along with Block point, uh, love John Merchants. And next, you know, kind of take us through what tipped you off, what made you start, you know, digging in, what do you know, what can you share? Um, and, uh, again, great job by you and the team for being so supportive, uh, to the channel. So, floor's, yours, John. Oh, hey, thanks so much.

Yeah, this, this, as we were joking around with our amount of sleep over the past couple days, this has sort of been kind of an all hands on deck effort. Um, burning the candle at both ends here just to, uh, our, our effort has really sort of been across, I guess, four different, you know, avenues here. We, we absolutely want the education and awareness. We're trying to bring this information as quickly as we can to the community, uh, and over at Huntress, right?

Our, our threat hunting and threat intelligence capabilities. We, that, that's what we do. We say hackers are gonna hack and huntress is gonna hunt. So whatever technical details, whatever information we can share, we'd like to help make that known to businesses and organizations affected, uh, and response to recovery, right?

That's, that's paramount at this point is being able to get businesses and operations back in action now that we're amongst the real work week, even if we had that holiday off. Uh, and of course, community collaboration.

So when we were flagged this, I just, I would, I would have to say around noon on, on Friday, July 2nd, uh, we originally were getting reports from a handful of MSPs, uh, and we wanted to validate, we wanted to check in, but we got started with a Reddit thread and a Reddit post that we have been actively in, in an ongoing updates. Again, sharing new information from the technical side of the house and the implications kind of what this all means. Uh, you could follow the timeline there.

From update one to two, to three to, i, I, I believe we're up to 14 now. Um, we are wanting to share just about everything that we know, uh, give the best recap we can, as well as bring to light, Hey, how could you survive a storm like this? How can the companies, and if, whether you're affected or not, whether you're impacted or not, how do you respond to a mass ransomware incident with that?

Uh, so we're, we're hoping to maybe have a little fireside chat, uh, with threat ops, team members, security researchers, and our hunters founders at 1:00 PM today. Uh, so we're hopefully being able to light off some of our own fireworks in there. I think we have a, a, a speculated proof of concept to really see this thing in action and understand what it does. Um, with that, uh, it's been incredible to see the absolute outpouring of community support.

If you were to look through our Reddit thread, or if you could even see some insight in our support queues, we have MSPs and businesses saying, coming outta the woodwork and saying, Hey, I'm over in New Zealand. I want to help. Hey, I'm in California. I wanna help. I'm in Mexico. I wanna help, uh, all over the world.

It's been a global response and an incredible, heartwarming and fulfilling thing, uh, to see everyone saying, uh, whether I'm affected or not, we have people on tap where people ready to get an arm in the fight. Uh, and, and that has been the best silver lining I think that, that we could see here.

I know it's e easy in a security incident like this to kind of crack down on some of our faults, some of our weaknesses, but the most uplifting thing here has been seeing the positivity and, and bringing a community Together. John, do you, do you wanna share a little bit or what, you know, kind of a prelude to the attack, um, that you guys are gonna be talking about? I, I can, I can tease a little bit, if that's all right. Um, yeah.

So we, we have been, we, we've been trying to stay engaged with this. We've been trying to be in the trenches. We've been trying to be fighting these fires just along with everyone else. Uh, so we're chatting with some of the other players, right? Um, and we've discussed FireEye. Uh, Tru SEC has been putting out incredible other intelligence and research with us. Uh, everyone is in the mix on this, um, from some of the tools from detection that we've seen.

Uh, Tru SEC has just recently put out one that seems to be a bit more thorough and looking for indicators of compromise, things that could follow from this attack. Uh, and we both seem to be aligning on the thought, and again, speculation is the right word.

Although we have moderate to high confidence in kind of what we're seeing from our own analysis, from seeing logs, from seeing artifacts that, again, the community out of the goodness of their heart outta the graciousness of them, has been willing to send in to better understand this threat. Um, we have the thought, okay, maybe are, are we seeing some authentication bypass? Are we seeing uploading potential files? And are we eventually seeing code execution?

Um, the technical details all beyond that, and the artifacts and the indicator of compromise are again outlined in our Reddit thread. If folks are interested, uh, I'd have to say, this is what we believe, but you have to take with a grain of salt, right? We, we all know this is still developing. This is still ongoing. So more information to come. As always. If I could ask you a quick question, John.

Um, certainly think in your, in your updates, there's a, there's a couple hypotheses about the attack, which you said yourself are not validated yet. Some of them are, some of them aren't. Um, one was that there was an authentication bypass that allowed them to enact the SQL injection, but then your analysis of the JPEG file, uh, indicates that there was potentially another type of injection as well.

Do you, do you have a, a more clear picture about the different vulnerabilities and, and the kind of exploit chain, or are you still kind of theorizing about that? So, I'll get nerdy here if that's all right. If you don't, if you guys don't mind me geeking out a little bit.

Um, we, we've seen it, the series of requests, the, the subsequent calls from these strange road IP addresses, potentially in AWS or Amazon Web Services where they would reach one a SP file to another DLL to another a SP file, uh, that ladder, that last most web file, uh, examining that and analyzing a little bit more, it, it seems to have that potential for SQL injection.

And again, that's not to say that is the vector, uh, also present in that are some other function calls and some other processes that could potentially lead to just code execution and command injection from what would have been uploaded with the previous DLL that could have been, could have been offered.

Uh, and I'm trying to be politely vague where I can be, um, and, and I, I am leaning, I think we and the team are, are, are more leaning towards that, okay, strictly code and command injection, not strictly SQL injection. And we're, we're gut checking ourselves on that. 'cause I realize as you just said, Hey, our updates, we were, is this SQL injection? We're questioning, uh, we, we still have to develop and analyze and learn more as we go through this.

But again, our priority has been just sharing that intelligence, that information, what, what we can see so MSPs can kind of better prepare and better understand what really is here, what are these artifacts, what happens when, et cetera. Uh, so I hope, and it's our hope, right, that this is something of use to you in the community, because when we see that community coming together, sharing this information, uh, I think that's the best way that we can really get through this. Yeah.

Uh, Did I kinda get to your answer there, cor Ryan? I'm sorry. Yeah, Yeah, I think so. I mean, I think it's, you, you're, you're trying to be polite. You're not trying to front run what Kaseya has to put out, but you're saying there's a certain way that this had to have unfolded and it's very clear that the type of injection that we're talking about very likely would've had to bypass authentication.

And then there is very likely two different types of injections given just the files where you were seeing dropped in the order of execution of things within the attack. Is that a fair summary? That is fair. Uh, and again, if you'd like to get nerdy and still get technical with us, we're lighting off our own holiday fireworks for our 1:00 PM webinar today, so, Yeah. Um, so one more question for you, and I think this is an important distinction for MSP community. They may not understand this.

Can you talk about the difference between a zero day and a one day, because this is being referred to a lot as a zero day, but it's actually not, it's actually a one day. Can you help people understand what that means? Yeah, I, I'm, I'm glad you asked that question, and thank you for, for kind of having the opportunity to chime in with, with my thoughts, right?

Uh, a zero day is given the term zero day because us good guys, the defenders, us and the blue team, us trying to protect and prepare for these cybersecurity incidents have had zero days to prepare, as in this is totally new, it's earth shattering. We'd never seen this before. Uh, and the obviously kind of mirrored the opposite of that, right? Is when you, you are notified and you have an inkling that there could be something afoot.

Uh, so when, when we say this is a zero day, uh, I kind of bite my lip a little bit. I, I don't know, uh, I don't, I don't, I'm not sure, um, that is the definition of, hey, zero days to prepare. But if we are potentially hearing word from maybe DVID and other researchers may is, is is there mort to a story here? Right? And it's also fair to say is perfectly possible that given there was a ladder, as you described, right?

A, a a series of exploits that occurred, some of those exploits could have been zero days, other of them could have been one days, right? Certainly. So we still don't know yet, but, um, it, it is becoming somewhat clear that some of these things were known and in the process of being mitigated. So it's not totally a zero day.

And, and I think it's, these things are an, an important distinction for MSPs to understand as they're thinking through how they ingest this information and how they assess their own risk profiles. I know the question came up is how similar this is to the prior SolarWinds attack, and I would say it's less similar with that and more similar with the exchange proxy log on, because the exchange proxy log on was very similar in that it was multiple vulnerabilities.

There was, there was, there's evidence that there was knowledge ahead of time. So I, you know, again, I I think that's another one that got called a zero day that I don't feel is I bite my lip too on that one and the same as I do with this one. Yeah. And the, and the one day also is interesting, right? Because we're talking about the one day from the perspective of Kaseya, from the perspective of an MSPs, this was a zero day, you had no time to prepare, you had no time to protect yourself.

So again, there's some nuance in, in how you're gonna, as, as this gets communicated about, right? One of the questions you need to ask yourself is, from whose perspective is this communication coming from? Is this coming from the perspective of Kaseya? Then, then the terminology could be very correct from their perspective, but given your supply, their supply chain to you, that means a different thing for you.

And so you might disagree with some of the terminology, but that's really because it's, we're dealing with supply chain, it has to do with the perspective of the individual within the, within the, the attack. I would, uh, I would, I would echo if I may, uh, yeah. When we, when we keep discussing supply chain, I've had conversations with other researchers and nerds and geeks like me, right? And we said like, uh, is this a supply chain? Like, like SolarWinds is a supply chain?

Well, it's, it's not the CI ICD continuous integration, continuous, it's not the technical pipeline, but this is certainly the procedural pipeline from VSA to MSP to SMB, et cetera. Uh, that is, I think where your point exactly, Ryan is, is so important is that there are sort of three prongs to this, and it's a matter of perspective whether we have accurate, precise, a thousand percent certain data. Okay? So, um, Derek doesn't trust me because I work for Datto. Um, yeah.

And I don't think hunters has raised any money. So, which is, which is fine, which is fine Report to anybody either. So, and that's totally fine. And so there's a question that I've been sitting on, which is, um, uh, I think Brian Weiss asked it, and there's a, there's a thread of this of like, are on-premise, RMMs not safe to use anymore. Um, and I don't wanna answer it because I, I am an RMM vendor, right?

Um, and as I'm thinking about it, right, there's a question of, it's not really a question of cloud versus on-prem. It's a question of did you have the right defenses in place to prevent malicious execution and access of your system, right? And so the question I've had is, if I'm an MSP and I had ip acls, right? Network level ACLS restricting access to my VSA instance from the internet, would I have been compromised?

So if, if I only have, you know, 10 IP addresses in the world that can connect to my on-prem VSA instance, or if I, if I somehow limit what can talk to my VSA instance, do you think that that would've been sufficient to prevent the exploitation? Here, Here I would have to add a little asterisk and have the weird legal speak. It depends in a, in a, in a horrible, crappy answer for you. Uh, I, I guess I can, I can, I can say this again.

If we were looking through the potential web files, those ASP files at the technical level, um, one of the very initial ones that are accessed has the functionality and, and, and the code check really that at the source code level determined what will happen if this is a cloud service or if this is an on-prem service.

Uh, and that is where there is that specific distinction why on-prem is affected and why cloud is not, um, if you were to limit and narrow that attack surface, say oh 10 x, y, z, whatever criteria to be able to access that, uh, you've, you've, you've done that, but that capability is still there. Uh, I think this does bring up the new conversation of accountability, right?

Uh, 'cause now that we're, uh, we're seeing these things now that this has sort of been brought to light, does it, does it raise the eyebrow a little bit more on the technology and the software and the solutions that we rely on, uh, that we know are administrative access godlike powers? It's a C two framework and it's command and control. Yep. So yeah. Yeah. Raise think the biggest thing is there's, uh, the conversation that is on-prem safer than cloud. RMM is not the right conversation, right?

It's a conversation bound. Is it, is it secure? And what are my responsibilities in securing that? And so the question is, if I had had network access restricted to VSA, would I have been in a better position than, um, you know, than someone that had it just open to the internet? And the answer is kind of, maybe it depends on what your access list looked like, right? Um, for that. So Andrew, go ahead. If you pulled the cord and turned it off as, as we know.

So, So, so John, um, awesome update and thank you so much for joining us. I'm gonna bring John Merchant in on now and, uh, yeah, really grateful that, uh, you guys were able to come on and all you're doing for the community, um, John and or if, um, you know, your team's in there, Andrew, I think Andrew's on there, feel free again, 1:00 PM if you want to get into the technical deep dive with John and team, uh, feel free Andrew Kaiser to put that URL in there again.

Um, and, uh, look forward to seeing you under different circumstances. It was great catching up with you in Orlando. John, Thanks all. Great to chat with you. Thanks John. Okay, so let me go get, um, John Enson on up here with us. We'll do, we will chat with John and then we'll lead into, uh, Chris a little bit about your perspective. And then just let me know, Chris, I think you said, yeah, if you could send me that Jen's name again, that would be awesome.

Uh, you know, so I can keep it's in, it's in your text. Okay. Fair enough. John, welcome. Hey. Hey, thank you. I hope you guys can hear us. I took the opportunity to bring Javier in. Uh, we dropped our normal, uh, setup here and went old school so we could fit both of us On Andron. Man, this is great. He's better Than Juan. Yeah, exactly. Exactly. So, uh, no, I appreciate the opportunity to, uh, come on chat with everybody. Yeah.

Well, John, I know you and your team were very early on in this as well and some of the first Reddit posts and things of that nature, you guys have been at this all weekend, appreciate all you and your team has been doing for the MSPs. Um, maybe same kind of, you know, question to you that I put to John. You know, take us through chronology, what you can share. Obviously there's things you can't Yeah.

But, um, you know, what, what kind of, you know, got you your guys', uh, you know, detection up and, uh, made you start really digging into this. Yeah, without a doubt. That's why I brought Javier Javier's, our vice president threat ops runs our whole 27, 24 hour, you know, operation here. First off, you know, before we jump into like the, the, the kind of play by play, 'cause that's what we wanted to give you guys a bit of a play by play from our perspective.

I wanna give a shout out to m ms p Geek MSPs r us on Discord s next door, right? They're our next door neighbors and XNSA guys like us. Um, getting out that word that early in those kind of forms like Reddit and those channels, you know, before we even got our own kind of social media posts out there, we actually posted on Ms. P Geek and then we're over in Discord. And I think, to be honest, getting the word out to shut those servers down, uh, was incredibly important.

The other thing I'd like to say is, you know, I, I totally feel for all of the, the victims of this breach and, and k say as well, this could happen to a lot of cloud companies and on print, right? If, if you're connected up to this bad neighborhood called the internet.

The one move, and I'm gonna kind of highlight why in a little bit, I think that was really important, was the speed with which Casa pulled down their cloud infrastructure, um, to be honest, because it actually allowed us to more narrowly target a response to make sure the rest of the on-prem base couldn't get impacted.

Um, and, and I think, uh, Ryan, we, you, you asked a, a question on kind of the network access restriction, you know, big picture, this is, I hope we didn't jinx anything we had the day before a threat meeting. 'cause leading into holiday weekends, we always see a big uptick. So it's kind of like we gather, gathered the troops around and we literally half made a joke about, I hope there aren't any RMM exploits. RMM exploits are literally the worst case scenario, right?

Because I think, like John Hammond said, it's basically, you know, it's a wonderful, you know, backdoor is the wrong term, but you have a piece of software that can basically execute anything at once. And so when you can hit their command control servers, it's, it's incredibly fast the time to booms really quick. Um, and so, you know, the, the speed in getting the, the word out there and that kind of info sharing is, is step one.

And then step two is actually taking, you know, proactive action to get these things offline. So I'm gonna turn it over to Javier. You can lean in here a bit and kind of give us a play by play on the timing of what we saw from our side and why, and how we figured out, uh, oh, this probably is an exploit, you know, so turn it over to your ex and I'll jump in in here. Yeah, absolutely. Lots of back and forth. Yeah, absolutely.

So yeah, I wanted to give an update or kind of recant the story of, you know, someone, uh, you know, on the front lines, uh, getting prepared to respond to something that, you know, like John said, we just talked about the, uh, the day before and we kind of always had in the back of our minds and always we're prepared to, to deal with, you know, so, you know, to, to kind of start things off here, the typical ransomware scenario that we see out there daily is, you know, someone compromise their credentials.

They get on a machine, they start spreading, they start scanning, they make all sorts of noise, uh, and all sorts of other types of, uh, activity that you can kind of get 'em out there early and get 'em off. And way before it's even attributable. Uh, this is 99% of, uh, the, the incidents we come across just turn, you know, pretty much light incidents and events or even recently, like with the print nightmare, uh, situation, you know, we can able get detections on it.

We get it into our lab, we figure out, you know, how the data it's causing and ways to respond to it. Uh, and it's also something that someone has to dwell internally a little bit. You know, you, you, the hacker has to get in, has to get, you know, at least some type of credentials to launch this print nightmare exploit that we've all been hearing about, which is exploiting the print server. Um, so there's a lot of lead up into that.

But on this side of things, you know, when you're exploiting something like an RM tool, um, there's, there's not much lead up to it. That's why it's always been kind of a, you know, gets the pit in our stomachs on this. And so it was around, it was around, so we had, we had two simultaneous calls going on that, you know, uh, unbeknownst to each other were me, myself was on the phone call with somebody.

'cause I just, I heard something odd about Kase and I was like, ah, you know, let me take this personally. And then my other one, the director heard something odd about K also at the same time was like, let's take these calls simultaneously. And that was about, you know, 1:00 PM Um, and so they, and so from one and two, we were on these calls and all of a sudden we're starting to type data, you know, we use, uh, you know, uh, chat systems and we're, we're started typing data back.

So like, there's something weird going on at the, and exact same time, my director side, uh, messaged me. He goes, there's something weird at K. And that's when I was like, that's when that, that's when the jaws dropped, my eyes lit up and I was like, no. And, and I, you know, the wonderful gentleman on the other line who was, you know, just dealing with the worst scenario, uh, you know, gave me just a little bit of information to where, you know, I started getting that pit in my stomach.

Like, this is, this is bad. This is likely something wrong With Kaseya. Um, yeah. So Javier, would you say like, our first call is at 1259? Exactly. Oh yeah. That's where we had a customer say, Hey, we just saw a ransom note pop up. Right? And then, you know, we're starting to investigate. And what's really unusual, like, like Javier said in the beginning of these breaches, most all these ransomware include tons of indicators before you get ransom. Like mass share, dollar sign mounts, right?

Failed or successful. Like, you see that all the time. In this case, there was nothing. And we see nothing that's usually indicative of an exploit. And this one, you know, unfortunately was the worst case scenario. Uh, and then was it 1 32? Exactly. We had, uh, another MSP partner, you know, calling, so we're going back and forth and I would say from a big picture, we had, um, several things going on one, we, we realized it was Kaseya.

We were damn confident it was an exploit base because of the speed with which it, it, it was, it was deployed out. But the question was, we had, we had only two MSPs affected, uh, in our customer base. 'cause we're able to knock all the other ones offline so quickly that they didn't get exploited. Um, and, and so the, uh, and the only two are on-prem, but no, at this time, discords lighting up slack's, lighting up is, is is doing a great job over on, on Reddit, Reddit's lighting up.

It's clear what was not clear to us at the time. So this is kind of that kind of fog that you get in the middle. Is this affecting their cloud as well? And so, to my point earlier, I would say Sade, 'cause you know what we are actually prepping to do, we have a platform behind the scenes where you can kind of get, take an IOC and see every single host name and you know, where, where this is running across our customer base. We are prepping something we've never done before.

And that actually isn't native to our platform, which was a mass response across every casee endpoint agent in the network. Or, you know, John, just quick question. How did you, how did you do that? What was it a block of some type? Just curious. The technology To be honest.

What we were, what we were planning to do, and thankfully we, we didn't, 'cause it wasn't necessary, uh, is, you know, obviously the Kase RMM agent, if we could snuff that guy out and blocked the command and control, I think it, you know, it wouldn't be able to, to execute that, that payload. So we were considering that because we were picturing in our minds, oh my gosh, every single Kaseya customer of ours, which is hundreds and hundreds, is gonna get ransomed any second.

And there's almost, you know, assuming it was cloud, but luckily it wasn't. And can say it took that down really fast. We were prepping to do it. So we had part of our engineering team getting ready to deploy and update to allow us to do this. 'cause we had A-A-A-A-A framework and a pipeline that we do something similar with that we could have repurposed for this. So kind of a out ofAnd response, you know, kind of technique.

And so what we, when we heard the cloud was taken down the exact and, and then, and then really Javier's director, we're all on conver. We have a big, when something massive like this, which has been impressive, we have a little crisis committee. Yeah. You deploy the crisis team quick. Yeah. And so it was one where Will says, well, you know, John, the only in Javier, the only two customers we've seen were OnPrem and they were kind one build back as well.

So we'd also don't know, maybe Huntress knows, you know, is there a specific version, you know, of the on-prem server that was impacted or not? Like we don't have a big enough sample set to, to see that. But um, so what happened then is we said, ah, forget the cloud. We're not gonna do this mass response. We immediately did a query, found every host name in our customer base that was running on-Prem vs. Uh, VSA.

I saw about five of 'em had gotten our emails that, you know, 'cause we pushed out on social and, and everything. And, and so we did a couple things. We repurposed our whole business development rep team who normally smiles and dials and annoys you MSP, sorry about that. For, you know, to get a demo with us and said, here's the several hundred toe customers. Start calling every single one of 'em so that we could get mass calling out.

And then the other thing we did is we identified all the on-prem servers, and we have a feature in our tool, you know, not totally dissimilar to other EDRs where we can, we call it detained, but we can basically knock the, the servers offline. And so that kind of goes to Ryan's question, you know, big picture Armand platforms, you, you can't really operate as an MSP without 'em, right? I mean, you, you have to have 'em, but they are so unbelievably powerful.

You know, one model I do think, you know, obviously pulling the cord worked because none of those customers we did that for got hit. You know. But I think considering, you know, if you were all traditional on-prem and you didn't have to expose that server to the internet, right? I think odds were, odds are you wouldn't have gotten popped. The vulnerability still would've been on the service, but you probably wouldn't have gotten hit.

You know, I think as we move to more zero trust, you know, an application specific network access, really trying to figure out how do we get such a potent system one layer deeper so the services aren't sitting on the web. Because, I mean, there are so many cloud platforms I could think of. If there's a zero to exploit that took out their, whether it's cloud-based networking platforms. I mean, could you imagine if someone smoked all those routers and switches at mass?

I mean, that's terrifying to us. You know? So, and I agree with everyone else. We would've never called this a supply chain attack. A supply chain attack to me is one where you infiltrate it, there's two forms. There's kind of a hardware and software version. There's probably more, but at a high level, you know, one where you infiltrate a company, get into their code base, leverage, you know, an update mechanism, you're kind of inside the tent doing that.

The other would be more of a hardware interdiction where you're hitting a piece of hardware, hooking it, you know, putting some sort of, you know, capability in there. That's, that's when we think supply chain, we would con, to us this was a zero to exploit because we didn't, you know, we were scared of it, but we never saw it coming. And, and we had a respondent, you know, instantly.

So, you know, for us, you know, like everyone else at the 2:00 PM time, there's, you know, our initial indication was 1259. We found an an IOC actually that was incidentally collected at 12:53 PM So a few minutes, the, the MSP that called us actually was really squared away and did a great job and understood that some, some bad stuff was deployed from their, their Kaseya server.

And you know, for us honestly by, you know, 4:00 PM or our, everything was quiet and it stayed quiet through the weekend mostly. 'cause everything was offline by then. You know, you see a couple stragglers as a box comes back online that, you know, there were some indicators, but Awesome job basically. It wasn't that bad, at least on our side. But we know how terrible it was.

You know, big picture for the MSP community, obviously, you know, it's tough on the trust front too, with the customers of an msp. So this is something is a, it's industry. We have to figure out how to get a lot more vetting and judging of what gets deployed out of an RMM platform and execute it on the endpoint, you know, and then reduce the attack surface if you ask me by trying to make it, you know, harder to get to the web interface, for example.

And we start platforms that, 'cause otherwise it's just a matter of time until there's another sport on another platform. I mean, it's just how the game will go. Sure. So, um, really appreciate it. And, and John, both you and Javier, um, Wes, you've been quiet. Any and anything you wanna ask John Javier, not that you have to, we have, uh, coming up still Chris and Chris is gonna bring on an MSP and kind of talk through the IR side as a next step for everybody.

We're actually gonna bring somebody on that was impacted. Um, but Wes, any thoughts? Well, um, maybe nothing specifically for Black Point, but I, I'll ask it to you John and team. Anyway. Um, so one thing that I, I find myself coming back to a lot from this entire conversation is, I, you guys have a military background, right?

So you guys are kind of used to this world of like, uh, the same journey that military, uh, whether they're contractors or just anything inside government sector, having legacy stuff all over the place, way older than, you know, SP rms that may be 10 to 20 years old. Way, way, way older than that.

Do you guys see a similar journey that, you know, the, the major RMM platforms are going down as well with legacy software and, and having to work through, um, you know, how important security is now and it didn't used to be five and 10 years ago. Do you guys see a similar journey there? Boy, I mean, I would say yes and no. Wes, I mean there, there's two. This, this kind of cloud infrastructure can cut both ways, right?

Like, so on-prem stuff, you know, I used to say a lot of times, sometimes it's harder, harder to target a single person at home when you're trying to find one bad guy and get after them and hook them than it is to hit a company that has, you know, an engineered network and you know, people online that are telling you they're the domain administrator for a company or whatever.

So I think in a case like this, you know, maybe the on-prem legacy, maybe no web application firewall type stuff made the on-prem stuff more susceptible. Um, it probably also made it less widespread because I could flip it on its head and say, you know, as we throw more stuff into the cloud, right? And we put all our, can we turn the cloud down fast? Yes. But if that cloud got popped, if the build infrastructure, could you do way more mass damage? I would say, yeah, I would say yeah, for sure.

Um, you know, and, and so I think at the end of the day, it really has to do with, there, there's several things that go into it, right? And obviously having regular reversing against your own platform and pen testing, but really reversing even like a step further, like attack those platforms. Like you were a nation state trying to develop an exploit for a specific component, right? I think that's important.

I think trying to reduce the attack surface by not putting such a potent system, like reducing how much it's actually accessible from any IP on the internet. I think that's another key area. Um, and, and then, and then, you know, the basic stuff we all, we all talk about for more of your on-prem infrastructure, having MFA, I mean, the vast majority of our customer base still doesn't have MFA and VPNs, right? I mean, it's John, it's craziness to John.

Can I just jump in because I love what you said a moment ago, last week we did a threat modeling workshop. Irony of ironies. It happened to be, uh, gold Southfield, which is, That was on, that was last Tuesday. Yeah. So, but I love what you just said about taking it through the testing and, and Ryan talks about this, I add nauseam, but thank goodness, uh, know your, you know, you've gotta know your enemy.

You gotta know your, you know, what your threat profile is and then be able to, you know, test against that. And, and I'm just, I'm really thankful you brought that up because that theme is starting to hold some serious weight and, and we hope, you know, simple controls like you just pointed out, you know, MFA, you know, I mean, um, so yeah. Anyway, I just wanted to point that out real quick. Yeah. I'd say the, the last point on this is just operating with an consumer breach mentality, right?

And having like a response. 'cause you know, I really think in hacking, like there's one side of the coin, you know, where you're all dealing with malware, right? And how that malware gets executed, how jump up code works. But there's a whole nother part to a lot of these hacks.

This one is a little unusual where, you know, they could cut past a lot of this where it's very behavior, we'd call it trade craft, but behavior, how you figure out where you are in the environment, how do you spread, how do you answer all those questions? Where on the bad guy side, that's actually a time point of vulnerability. That's one of the areas we focus on a lot. And then merge in anti-malware style detections.

I think that's a pretty effective way too, but it's part hygiene, you know, it's, it's part continuous to, let's see, testing your infrastructure, especially if you're a cloud provider or have agents on people's computers, right? I mean, that was the one common part of the SolarWinds is they had an agent on the endpoint computer, the rest was not in common. Um, so yeah, I hope that gives everyone a little bit of a, a window into just how this unfolded for us.

It was, uh, yeah, the pit in the stomach. When Javier calls me, I usually get nervous. He doesn't call me very Often. I usually call myself the, I never have good news. It's no fun. So I know we're a little past the top of the hour here. Um, if it's okay with the team, John Javier. Um, thanks dudes. That was awesome. Look forward to having you guys back here. Great. John Javier.

Um, I know, uh, Cody's been more than patient trying to get you guys on and, uh, I'm really glad we have an opportunity. Like I said, you we'll be having more you with us. Um, I'm gonna ask that, uh, I think it's, you are in as Eric Wood, but Eric Woodward are you in as Eric Wood, if you could just maybe say something in chat for me and I'll bring you on here with Chris to chat a little bit. Um, thanks guys. Really appreciate it. And uh, uh, boy, this has been really, really cool.

Um, but let me just see is, I'm gonna try maybe pull Eric Wood, uh, Chris, I think that's who it is. Uh, but Think so too. Uh, although there's an, Uh, ProTech support, ProTech Support. Thank you, Eric. Yeah, We would've been pulling somebody else That've been, that would've been, uh, entertaining. Yeah. Okay, Eric, uh, fingers crossed here that, uh, okay, so let me remove those guys. Let me bring you on, Eric. And So while he's doing that, Chris, I have a quick question for you.

It, it appears as though sure, the attackers did not actually target destruction of backups during this, uh, exploit spree. That's different from what we typically see. Do you have any hypotheses or intelligence around why that is? So, um, what I've heard from others in the community that it wasn't that they didn't target it, they just did a bad job.

And so typically when you see, when we've seen this, so when we've seen these, especially this threat actor or this group attacking MSP directly, they're very thorough and we'll go after even segmented backups and so on and so forth. In this particular situation, since it was much more of a shotgun approach that they didn't get, uh, the backups.

Not to say that they didn't get some, 'cause we heard earlier that there was some backups that were on windows, probably on windows domains or some of that nature. Just ones that, uh, MSP should be ashamed of, of doing it that way, let's just put it that way, did get hit. Uh, but yeah, we think that, um, and, and again, behind the scenes and talking with other IR firms, it appears that overall there has been a lot of backups intact.

And that has been a very big plus, uh, for these MSPs in, in, in recovering their client's data. Now obviously, the workstations is the biggest pain point. I mean, they don't necessarily have backups for, for workstations, so a lot of their trouble has been really focused on, or their time has been focused on trying to figure out how to rebuild so many or re-image, so many laptop, uh, machines across their customer base. Thanks. Thanks for that, Chris.

Okay, Chris, um, let me let you set the stage first off, for those of you that may not know Chris, uh, Chris Lair, uh, EVP of solid security CFC response, um, regular contributor to the cyber call cyber nation. Um, and, uh, just overall really awesome, good guy to the MSPs in and of itself, uh, uh, known Chris now for almost five years and has worked probably more of the RM breaches than anybody in the country. Um, and, um, being CFC, now, the insurer, also a lot of their small businesses.

So Chris, I appreciate all you do. I know you've had no weekend, um, and often don't get much sleep at all these large scale compromises. So appreciate you coming on from the IR side and, uh, sharing some perspective with everybody and, and Eric, um, good to see you, but not under these circumstances my friend. And, um, can just check audio Eric real quick with me. Yeah. Can you hear me okay? Oh, yeah. Sounds great. Alright. Okay, Chris, so yours. Yeah, so we have, yeah.

So first of all, just introduce everybody. Eric Woodard. Uh, Eric is outta Salt Lake City. Uh, he's been, he's been an MSP for quite a long time. He's also been a Casee customer for quite a long time. Uh, Eric's been through the ups and downs, and he is gonna talk to us about that. You know, we were, we were hoping to have a, uh, at least one MSP that actually had their own, uh, Kaseya server that was affected.

But both those guys, unfortunately, uh, were overwhelmed with customer calls and couldn't make it. So Eric was kind enough to, to jump on. Uh, Eric is a SaaS user and he can kind of talk about maybe, you know, his kind of decision behind that as well. And then take us through some of, uh, uh, of what he felt through this process. You know, obviously the SaaS was shut down, uh, with little to no warning.

And, and Eric can kind of speak to what, what he did and how he kind of communicated with his customers through this. Uh, because his customers somewhat, uh, a lot of them are, are accustomed to him communicating during these types of situations. So I felt that his perspective would be extremely valuable to everyone on this call. Well, thanks Chris.

So, one, one, uh, the reason I kind of got pulled into this is we actually have a situation where our RMM was compromised, um, February of 2019 at 9:53 AM. Um, never forget it. My heart goes out to all the people that are dealing with this right now. It is, it's un it's unreal. Um, the other thing is, it's really hard for them to be on calls, um, speaking from their physician. They're on attorney calls, they've got gag orders, all that other good stuff.

So we have a situation where, um, you know, I don't wanna get into details and belabor it, but we, our RMM was, was popped. Um, we had two fa, we had all the things in place. We had, you know, loaded firewall, we had threat hunting, we had all the good things in place, and we got hit. And so I wanna emphasize it's not if it's when, um, we, we had our guards in place. This wasn't a LA CMSP story, right? Um, what they're going through and what Kase is going through.

I'll say Kaseya is handling this a thousand percent better than they did before, as well as I think ConnectWise handles things way better than they did back when we got hit. So, um, just wanna kind of point out how we got pulled into this. Um, you know, the couple things. So first of all, not having SaaS right now, yes, it is a huge dis me, um, feature, but I promise it's better than the alternative of having your customers hit, right? So, um, we met as a team today. Um, we're about $3 million.

MSP 17 people we met as a team, just said, Hey, what other tools do we have? Well, we have, well, we have different ways of doing things to, to assist our customers. We sent out a message, which I had blessed by the attorneys saying, Hey, this is what's going on this weekend. We have a print schooler problem, and we have one of our tools that have been hit and, uh, you know, be patient with us as, as we kind of work through this.

So, um, you know, I understand the heartburn and the frustration with vendors. Um, you know, it is not if it's when, um, and I think you need to prepare and look at your tools and determine, okay, if this tool gets hit, or when this tool gets hit, what are we gonna do? How are we gonna communicate? And so on. So, anything else you want me to touch on, Chris? Uh, no, just, uh, that, that, that, that's pretty good. But just kind of take it through, like maybe just a little bit about the emotions.

Uh, on Friday when the information was coming down and you were kind of in that little area of limbo, how did you kind of communicate with your staff during that kind of gray period and, um, and that type of thing? And, and have you had any type of, uh, clients that may be a little bit more difficult, uh, to deal with during, during this situation? Or have all they been understanding?

I think some of the MSPs who have never maybe been in this situation, we talk about IR a lot, maybe want to kind of hear some of the, you know, the dynamics of having to deal with different types of clients. And I know you do, you have clients that have different demeanors and different approaches and so on and so forth. Sure. So it, it's really an interesting thing. Some of your worst clients, we are totally understanding, and your best clients become your worst enemies.

So it's really hard to predict what's going to happen. Um, you know, as the motions come down, like it's hard to describe the shock that you go in as a business owner. Um, I was lucky enough to have Chris by my side. Um, I had a good attorney and they, you know, they kind of sat me down and, and virtually slapped me and said, Hey, look, pull your s**t together how you react right now as you determine how successful you're gonna be at the end of this. Right. You know, silly things.

Eat three meals, take a break, walk away. I mean, I had a phone to each ear, customers on one attorneys, insurance companies on the other for nonstop for almost a week. It took two months, 24 7 to get us out of it. I'm just barely dealt with my, my last like, legal case regarding this, right? So it is a marathon, it's not a sprint. Um, you're going to go through shock, like as a business owner or even a management team. Um, I can kind of see it in some of the weary faces on, on is call today.

There's a lot that goes on and there's a lot, uh, of pressures that you've probably never had that you need to kind of work through. And you just gotta kinda keep your, your wits about you. Um, you know, how you lead through this is how you're gonna survive another side. The other thing, so This, yeah, real quick. Uh, so this weekend, specifically Friday, your, your clients. Did you have any more questions or were they, they, they somewhat, did the messaging go over, go over?

Well, You know, so great question. So when we had our event, we lost about 10% of our clients, right? They're used to us being very transparent and very open with them. And so when we saw this stuff going down the pipe, we're like, well, we gotta communicate with them. So we drafted up an email, sent it out through HubSpot so that we could track who opened it and who didn't.

We saw openings all through the weekend, and I had a number of email, uh, clients come back saying, thank you, I know you got our back. Thanks for that communication. So it's a big deal to communicate this stuff that you're aware of it. The last thing you want is them hearing about this on CNN and then coming to you saying, Hey, you know, what are you doing about this? You even know about this? Or are you on vacation? So, you know, over communicate.

Um, we're probably even gonna do a webinar of, you know, what to expect with this kind of coming up with our clients. Um, the clients that, that we retained, I mean, we're tight, right? They, they take our recommendations. Um, it, it's a, there's a little PSTD, uh, and, and no, you know, disrespect to any military here, but there, this brings back a flood of, of bad, you know, vibes and, you know, you gotta keep your head up and get through it again. All right, thanks.

Anybody, uh, Ryan West, Gary, anybody? Andrew, have any questions for Eric? No, just Eric, I appreciate, I'm gonna let you guys, I, I had her go inside. Go ahead, Gary. No, I just, Eric, I appreciate you coming on and, uh, unfortunately, you know, I've had this, I've had conversations with many people, right? Uh, over the past few years in, in the same boat. And, um, you know, not all of them get through it the same way that you did.

The ones that do ended up being a better MSP with better client relationships. Um, and it changes them. The idea we're trying to do here is get people there without having to go through that experience. So yeah, we appreciate, appreciate you sharing it. Learn from your peers. I mean, don't make their mistakes, right? So, um, yeah, Eric, I, I do have a quick question. Sure. Post that original breach, that incident you had, did, how did you change, you know, your approach to security?

Did you, did you take frameworks, risk management more seriously, hygiene, et cetera? You know, I, I I, I, it's kind of a funny story. We were probably ahead of the curve with security. Um, we, we were, we were preaching it. We had a, a, a graphic where we took our clients, talked about a wall, and how high do they want the wall? And, you know, we were ahead of the curve. A lot of the more advanced products in place, um, had some of the, the process and procedure in place.

So the, you know, the irony that we got hit was, was pretty rich. Um, the thing that I'll say is going through this as a staff with clients, like if, you know, it takes us five freaking minutes to log in anything anymore, right? Like, we have to into this and this and this to get this password, then we have to, you know, like, but you know what, no one on my staff ever complains. And maybe beforehand it would've been Simon and groaning anything to do with security.

Now the staff we just embrace and we just go for it, right? Um, we try to help other MSPs. I'm like a counseling center for the poor MSPs that have been hit. Um, you know, a couple, probably like three things. If I could list the kind of takeaway messages I learned from, or, you know, lessons learned is, um, use a d different RMM for your BDR devices than your rest of your servers. Okay? It sounds ludicrous. We're supposed to standardize everything.

Use a different RMM, don't whitelist anything. Okay? We had, I'm not gonna name throw 'cause I think that's counterproductive, but we had a very high end antivirus, you know, next gen stuff, and it was bypassed because it was whitelisted. So we're with antivirus now, but we don't whitelist anything we refuse to white list. Um, the other thing, and again, not trying to vendor bash here, but test your backups, but test at scale.

So we tested one customer, one file, one this when it, when it hit the fan and we had to test, we had to bring up a lot of servers. Our vendor fell flat on their face just completely. It was, it was a, it was a mockery, right? And so, um, you know, test at scale. So we should have tested bringing 300 servers up in a day before this incident happened. We had tested with a dozen servers. We had tested with this and that. But, um, yeah, that's, that's the, uh, other thing.

Then the last but not least, be transparent with your customers, like the way to say and ConnectWise handle things now 1000% better than the way they did before. Yeah, you're talking our language. Ryan, uh, resilience tabletops, Wes, Chris, um, you know, uh, Ryan, anything you want to add there? I saw you shaking your head on at scale testing and people process side, not the technology. I, I mean, I, I completely agree.

Like I, the amount of MSPs that actually test full recovery of multiple customers, even when they test the software as part of their diligence is like, you can count 'em on one hand and maybe a dozen of our customers a year will call in to do a massive, we have a team we call code RED Dr, that you can call in and schedule large DR tests with. The amount of MSPs that do that is, is rare.

And what you don't realize by doing that is that it, any backup and recovery strategy that you're going to use where you need to recover hundreds or thousands of workstations is not going to be fast. And so you almost need to understand what, what the timeline is for you to recover in order for you to even be able to communicate appropriately with your customers. So you should be doing that DR testing, you know, of your backups at scale. A hundred percent. I agree with that.

And, you know, if, if you test it with DA and, and you know, something is wrong or something breaks, like let us know. We have a responsibility to fix that. We need to be there for you. And, and it's the same with every BCR vendor out there. That's excellent. Great. So should we, good point. Write that down. Test that scale, make a note. So again, sorry for the background noise, guys. I had to go inside. It was like a zillion degrees outside and hopefully my power's back on.

Um, but I think, you know, in, in interest of time, Chris, do anything you wanna round things out on the IR side for us And Yeah, I think, yeah, some, some high points here. So, you know, we heard from a number, so again, not to echo information went out really quickly. So we've, we've seen a bunch of people talking about the different forms and different channels and modalities. And so I do think that that saved a lot of MSPs.

Uh, I'm hoping that sometime in the future that, you know, somehow we can do some research behind that to kind of understand how that did work and how effective that was. Um, I think, you know, from an MSP perspective, there are some lessons to be learned. I mean, we had some MSPs that thought they had cyber C coverage and didn't, and so they found out the hard way, unfortunately. Um, and so, but they, they, but other MSPs kind of stood up and helped them.

I mean, there was one individual who, uh, just to say he was, uh, frazzled might be an understatement. Uh, he was very, very overwhelmed. And there was another, a number of entities, uh, MSPs vendors and so forth that came to his aid as well. And, and that's very important.

And the other thing I'll talk about is, is, is, is I will share with this group that, uh, from the get go, um, as everybody that has seen me on these calls that I, I, I, I do have to deal with ransomware quite a bit and in this group the most, because they are the most prevalent, they, and quite frankly, they are probably the most effective of what they do. Uh, and sometimes they're too effective. They do stuff that just doesn't make any sense.

Uh, if you guys had seen in the news a couple weeks ago, uh, they are now starting to encrypt, uh, basically VMware, uh, VMs, uh, which they didn't do before. So now you have to get a Linux decrypter to, to handle that. And so that's very new and they, they continue to improve. And, and one of the things that came up with these, this group is I think that, um, they probably saw this as a, a big win for them. Uh, they had high aspirations.

Uh, you know, right now that that price out there is $70 million for a, for a super universal decryptor, if we wanna call it that. I could tell you that number was much, much, much higher. I think their aspirations were much higher. But that tells you they did have a revenue target for this. Uh, they had no idea how many companies that they attacked. Uh, they just knew that they had a, a they had in the thousands of endpoints.

And the other thing I think that, you know, we, we, we, we will need to capture too is, is there are a number of MSAs that use VSA, not, they don't use their own hosted version and they don't use the Kaseya SaaS. They use somebody else hosting Kaseya and providing it to them.

So I think that number of, of MSPs that was spoke about earlier, uh, will climb, because I know that there's some of these master MSPs or whatever we wanna call them, uh, that provide VSA were hit, and as a result, their MSPs were hit. So it'll be interesting to see how this all shakes out. You know, I think, um, I think from, um, an insurance and and attorney perspective, I think the preparation was good for this.

I think a lot of people were, were, were, were, were fearing this and somewhat prepared for this. And so I've heard things, uh, uh, uh, go well there as far as what people are saying that they're getting the guidance that they need and so on and so forth. But it's a long road from that perspective.

And, and I think that, uh, my hope as this does, uh, light some fires and some much, much needed, much needed spaces, uh, especially at the legislative levels, uh, law enforcement levels and so on and so forth, I can tell you that I've, I've talked to federal law enforcement and, and this one is, uh, uh, is, is being handled at a much, uh, higher level, uh, than previous attacks that I've had to deal with those individuals with. So, uh, that's kind of my thoughts. I am impressed.

Uh, I think this could've been far, far, far, far, far worse. Uh, so, um, yeah. So thanks for that, Chris. So in the closing few minutes here, I wanna give it to Ryan. Ryan, you and I spoke a lot over the weekend. You've been, you haven't slapped I know, and I appreciate all you do.

And I know you're gonna be doing a webinar for your partners on, you know, how you look at, you know, coding processes, et cetera, but you've, you know, been voracious in reading the, you know, the Reddit posts and you know, the Ms p gee, lot of good ideas, lot of bad ideas. Yeah. Can you maybe take us home on what message you want to kind of convey to everybody?

Yeah, it's, and it's challenging too, because some of the ideas are good and bad at the same time, depending on which MSP it is, right? So one of, there's three things I'm hearing a lot, which are kind of knee jerk reactions. So my first call to action for you is slow down. You don't have to make any decisions about changing your technology stack today, and you shouldn't. We need more information to understand fully what happened here.

And once we have that information, then we can start to ask our vendors more informed questions, right? Right. Now the question I'm getting is, would this happen to you? And I'm like, what is this? Is this a SQL injection leading to a ransomware attack? Is this a compromise of RMM? Is this, what is this? Like, it's not a well-formed question. And so we, we need to help the MSPs have that conversation, right?

And 'cause ultimately they need to have it with their customers, which is why they're asking us, right? But so slow down, number one. Number two, anything you are going to do as a result of this needs to be based off of a risk analysis, right? So Eric said, you should have your BCR vendor be different than your RMM vendor. That is both good advice and potentially bad advice. It entirely depends on the MSP and the vendors and the technology that they're using, right?

And so you need to understand how, what happened in this event, and frankly, all the other, uh, how attackers do what they do, the TTPs that we discussed in the Gold Southfield Summit last week, those provide you a framework to have a conversation with your vendors around what are you doing to protect your platforms from these tactics that these attackers use?

And having that conversation, if they can't have that conversation, or if the conversation is like, we're this 853 compliant, have a SOC two run for the hills, that's not the right answer, right? But right now, don't make any knee-jerk reactions. Don't bail on existing vendors. Don't. And like you could say like, oh, you're, you're a, you're a vendor. You, of course, you don't want people to freak out and move away from you. I want you to make the right decision for you from a risk perspective.

My fear right now is that people are making knee jerk reactions about what to do to keep themselves safe, and it might actually put them in a worse position than they're already in. Now, if you decide that you need to bail on one of our products to keep yourself safe, and that's a risk informed choice, I fully support that.

So I just wanna make sure that you slow down, understand what happened, understand how your vendors are handling it, what the risk profile actually is for you, have those conversations, and then make a risk informed decision. That's really all I wanted to say is that stuff like, you know, people are like, oh, I, I, well, I, I have this RMM and that's hard to replace them. My BCR vendor, so I'm gonna move my BCR vendor. And I'm like, cool.

So you're gonna go from Datto to something else, and the something else doesn't have cloud deletion defense and is susceptible to the built-in automation attacks against backups that are built into these ransomware variants. You did something that because it made you feel safer, not because it actually made you safer, because you didn't take a risk informed approach to that decision. And so we just need to slow down, get some more information, and go through a risk assessment process. Yeah.

You know that, Ryan, one thing I wanna add. We've been on this, we've been on this call for over a year. Okay. Um, with Ryan. Okay. Uh, Wes, when we started this call, you know, you weren't part of ConnectWise. I wasn't part of Kaseya. I mean, I, I don't think we've changed our, our view, you know, on this. And when I hear Ryan talk, I know a hundred percent he's not saying ever saying anything self-serving.

And if you think that's the case, you probably shouldn't tune in to this 'cause it wouldn't be helpful to you. But, uh, I, I couldn't have been more impressed. Go watch the three hours that were done last week on that threat modeling workshop. It's some of the best work, you know, I've seen. So yeah, I couldn't, I couldn't agree more. Um, okay. So again, apologize about the background. First, I wanna thank all of you for spending the morning with us. Um, this was a lot of time.

I hope you all found it worthwhile. I hope you'll come back and join us routinely Mondays at 1:00 PM Eastern. Um, I send an email, email from andrew@cybernation.com. If you could whitelist me that'll each week I try to publicize what we're gonna be talking about. Um, there's no sponsors here. I don't get paid to do this. This is complete thought leadership and we've had some of the best, most informative guests, um, as known.

So, um, lastly to the panel, to Mike Sanders from Kase to, uh, John Mond and Pru to John Meson and Javier from Black Point. Uh, really appreciate it, Eric, again. Um, good to see you. Uh, it's been a while. Um, you're one of the awesome dudes in this industry, so I really appreciate you coming on. Um, Gary West Crowd. Chris, I feel like you're a regular. Um, thank you for all you do and, um, for being on with us all throughout this. Um, and, uh, so I posted re's thing at one o'clock.

Please join them on behalf of Gary West, Ryan, Chris wishing all of you guys to be safe. Um, and we'll see you all Monday at 1:00 PM Take care, everybody.